> LLMs are vectorial databases
You use a bunch of technical-sounding words here to make it sound like you understand. But to be clear, nobody understands why the evolved weights of a NN make the decisions that they do.
Almost nothing is understood about the actual representations used for nontrivial concepts, decision algorithms, etc.
If you look at the field of mechanistic interpretability, compared to “GOFAI” like learned decision trees, an LLM is completely opaque.
I’m shocked how many otherwise well-informed people don’t understand or agree with this very fundamental fact of just how little we actually understand about why LLMs work as well as they do. They figure “it’s science, of course there’s math and theory behind it.”
AI research is almost as purely empirical as the gradient descent loops its practitioners use to optimize their models. “Why” anything at all works is barely an afterthought.
> I’m shocked how many otherwise well-informed people don’t understand or agree with this very fundamental fact of just how little we actually understand about why LLMs work as well as they do.
Well-informed people understand that LLMs work as well as they do for the same reasons as horoscopes, fortune-telling and homeopathy.
They work great for banging out POC apps or writing boilerplate code. This is a real use.
Homeopathy triggers the placebo effect. That is a real use.
If horoscopes and homeopathy can generate tests, then call me a Pisces and pass me the singular molecule of deadly nightshade toxin.
Be careful what you wish for. This weekend in tearing out hundreds of crappy tests that got vibed into a codebase.
...well, you could train an LLM on whatever horoscopes and then ask it :-?
> why LLMs work as well as they do.
That's a very different claim from being "poorly understood" though. The emergent properties of any system with billions of parameters is hard to understand completely, that's the fault of data science more than computer science or even mathematics.
I think ”poorly understood” is accurate. Understanding has levels. How brains think is also poorly understood.
I disagree, because you can represent the constituent parts of any AI model as code and data. We can reliably build AI with this knowledge, but not brains.
Understanding does have layers, and that's why "poorly understood" is a meaningless goalpost. A book can be well understood without researching the gematria behind character's the names when you write them in reverse. An LLM can be well-understood even if you don't comprehensively test each quantization for miraculous unexpected behavior at the FFN level.
A book can be poorly understood if you know it's made out of paper and ink, but can't read. The contents would be recognizable but are meaningless symbols and numbers. You might note that some patterns of symbols repeat, but be completely clueless why or what it represents.
Notably, you could still print them all day.
Being able to put a PC together doesn't mean you understand computers. If I ask you to make me a computer that runs a wristwatch, you'll likely be lost unless you have very specialized understanding.
I think we actually all know what "poorly understood" means. There's no need to play tedious semantic games.
The semantic game occurred when someone was equivocating "LLMs are poorly understood" with "brains are poorly understood", which are worlds apart in the extent to which they are "not understood".
“equivocating” is not the word you meant to use here. Probably “equating” I suspect. But yes, I stand behind my claim and I deny that it’s a semantic game. The two areas are not identical in human understanding but they’re on the same order.
If I ask you what makes a good fantasy novel and you explain to me the English alphabet and grammar, you actually haven’t explained what I asked.
It’s likely that at some point we will also be able to represent a scan of the human brain digitally as code and data. Assuming we don’t separately make a huge number of leaps in neuroscience which are far from assured, it will be very likely that we can get the appearance of an operational human brain simulated digitally well before we gain much if any understanding of why it does what it does.
All that is to say, being able to build something is not not not the same thing as understanding it.
Your DNA is merely data, and humans can reliably make more of it too.
If you are serious, comparing how well we understand the brain vs how well we understand LLMs, .. it's not a stretch simplifying that to "we don't understand brains, we do understand LLMs".
Because the extent to which we don't understand the brain, is quite overpowering.
Some people forget that when they say "but it's not different from what a human does" ...
I am absolutely serious. I agree with you that the extent to which we don’t understand brains is overpowering. And I would stand by the proposition that “we don’t understand LLMs in almost exactly the way we don’t understand brains”.
We know lots about human development and genetics and biology and evolution and neuroscience and the physics of how brains are connected and send signals and how generally they are put together and have names for their parts and all that, but we’re clueless when it comes to “the hard question” of how qualia and consciousness emerges from that.
The scenario with the spooky simulation of thinking that emerges from LLMs is in the same category, with different details. Lots of knowledge about the substrate of the phenomenon, little to none about the much bigger question of how we get the appearance of cognition from these trained artifacts.
Clearly we understand extremely well how LLMs are created mechanically. We invented them and are currently putting massive amounts of work into studying and improving them. But that work is perforce largely empirical; figuring out the why once again eludes us. It just goes to show how mysterious the underlying phenomenon of cognition is.
I'm not following you here, you seem to be conflating LLMs ability of language use with the brain's ability of thinking and cognition?
Are you saying that thinking and cognition requires language use? Cause I think not.
Or are you saying that language use is sufficient for cognition and thinking? Cause I'm also not convinced of that.
What I am convinced of, is that a machine capable of language use is capable of tricking people into believing there's a "there", there. In pretty much the same way as the famous supra-normal stimuli experiment made baby seagulls believe that a stick with a red dot was their parent. It's exploiting our instincts.
> as purely empirical as the gradient descent loops
Are you suggesting that gradient descent is an empirically found and not understood technique? It was originally proposed by Cauchy in 1847, its properties are very well understood.
You might be referring to properties of the domains its being applied to.
I’m not saying gradient descent was empirically discovered, I’m saying that its use in machine learning (or elsewhere, I suppose) is itself a form of empiricism in that what it is is essentially a repeated observe/measure error/adjust cycle.
> I’m not saying gradient descent was empirically discovered, I’m saying that its use in machine learning is itself a form of empiricism. A repeated observe/adjust-based-on-data cycle
The data is the input, the output is to generally find the lowest amount of a loss function. It’s a greedy approach because brute forcing is inefficient.
It’s no more empirical than a greedy algorithm for scheduling.
> It’s no more empirical than a greedy algorithm for scheduling.
Right, GP is drawing a distinction between search, ie mechanical exploration of a space, with understanding, ie having a map of the territory such that you don’t need trial and error.
> its use in machine learning (or elsewhere, I suppose) is itself a form of empiricism in that what it is is essentially a repeated observe/measure error/adjust cycle.
"Empiricism" implies that the technique is based on observable, but not mathematically proven foundations. If a problem space is convex, gradient descent is guaranteed to converge to a global optimal solution, regardless of whether you know the exact formulation of the space.
Applying it when you don't understand if a space is convex is another question, but that's not a fault of gradient descent.
> If you look at the field of mechanistic interpretability, compared to “GOFAI” like learned decision trees, an LLM is completely opaque.
I think the field deserves more credit than that, there are plenty of interpretability tools like
* natural language autoencoders for explanations of activations: https://transformer-circuits.pub/2026/nla/index.html (demo at https://www.neuronpedia.org/llama3.3-70b-it/nla )
* easier-to-interpret language model families like Backpack models: https://aclanthology.org/2023.acl-long.506/
* attribution graphs to trace internal reasoning steps: https://www.anthropic.com/research/open-source-circuit-traci... (demo at https://www.neuronpedia.org/gemma-2-2b/graph)
* functional analyses which have identified how LLMs do arithmetic - https://arxiv.org/html/2502.00873v1 - and how refusal happens: https://arxiv.org/abs/2406.11717
* data attribution methods linking training data to specific attention heads https://arxiv.org/abs/2601.21996
If we could give a comprehensive and global explanation of an LLM's behavior in a single paragraph, we wouldn't need the model to begin with, but that doesn't mean there's absolutely no understanding of the model internals whatsoever
>But to be clear, nobody understands why the evolved weights of a NN make the decisions that they do.
We might not understand particular "emergent" capabilities, but the low level mechanism is not just understood, but a deterministic algorithm with a handful of basic componets, that are well understood themselves.
> We might not understand particular "emergent" capabilities
The emergent capabilities are the only capabilities we care about
For allignment maybe.
For the core functionality and the optimizations we don't really need to know how the emergent capabilities decide on particular answers.
Which is why we could build LLMs before those features ...emerged for us to see, and why we can just code LLMs with the numerical NN algorithms we use, and do now have to go in and change individual weights.
Paraphrasing
We don’t? I sure would like to know!We do understand "thinking" though. That's literally the whole point of Attention is All You Need, the attention mechanism is what separates the transformer architecture from other neural networks. It's well worth a read if you haven't gone over it yet.
Features like chain-of-thought, long-horizon contexts and RoPE/YaRN all extend this thinking capability very transparently. The only remaining thing to study is the data and weights, which probably isn't going to contain some sort of miraculous revelation.
Attention is at about the same level of abstraction as spike trains or action potential, IMO. It’s a mechanism, it doesn’t tell you much at all about how actual concepts get represented. (It merely defines the substrate with which they can be represented.)
The entire field of Mechanistic Interpretability exists because just understanding Attention does not in any way help you to understand why a certain NN responds with a certain hallucination about a certain Chinese boat in this specific context.
> The only remaining thing to study is the data and weights
To me this is like saying “the only thing left to study in the brain is the connectome; probably going to be boring, we understand it already”. It’s almost all of the hard/meaningful stuff! It’s where intelligence and consciousness lives!
The entire field of mechanistic interpretability has gotten nowhere. The formalized, "causal" understanding of latent space conceptualization isn't any better than an LLM cargo cult.
It's wholly possible that you could study one set of weights for decades, and find nothing. There's no guarantee that any patterns outside of human language exist in that data. In this specific context, it's satisfying enough to state that [Chinese] and [boat] were both tokens in the tokenizer, activated by a feedforward pass through weights that favor [boat] after [Chinese]. There's not any guaranteed solution to this. There's not even any guaranteed problem; that hallucination is an expected behavior.
This is like saying “synapse firing is well understood” in response to “nobody knows why brains make the decisions they do”. Wrong level of abstraction for the question at hand.
Please go on, else you risk sounding like the person you’re criticising. The structure of the neural network is somewhat opaque because it’s hard to understand as the individual weights can’t be usefully interrogated, and naturally, it comes from big datasets which a human brain can’t really absorb in toto. Your comment was interesting so I’d like more of it.
They don't "make decisions".
That's like saying "my d20 decided to roll a 17"
But isnt the point that it did roll a 17. And no one knows exactly how (in the case of LLMs)? Therefor any description of the conclusion should be thought of as an anology. Decided, randomly accessed, etc.
It's actually no different for dice than for LLMs. Explaining accurately the reason for the exact outcome of any given dice roll someone makes would be stupendously hard. It would require lots of instrumentation and math and be poorly transferrable to another surface, another player, etc.
But even so people don't say that we don't understand how dice work.
Saying that we don't understand how LLMs work is exactly like saying we don't understand how dice, or tires, or golf ball shots work. Or like the old myth that we don't understand how bumblebees fly.
It’s different in this way:
The best way to model dice is the Physical Stance. You consider rules such as gravity, kinematics, etc. There is no “internal state”, “world model”, “knowledge”. If you prefer, in Friston’s terms, there is no Markov Blanket.
The best way to model a human is the Intentional Stance[1]. You mostly need things like beliefs, knowledge, biases, etc to build this model. In Friston’s terms, there is a Markov Blanket, an inside vs outside.
Without going into any irrelevant-but-interesting philosophical discussions about consciousness, I believe the intentional stance is most useful for modeling LLMs. Most of the success in predicting, debugging, optimizing these systems is in activities like understanding what they believe, what their intent was, what they observed, what they concluded from those observations. Also note that much simpler creatures benefit from the Intentional Stance; you will be more successful at modeling your dog if you think about what it “wants” rather than trying to run Physics on it.
[1]: https://en.wikipedia.org/wiki/Intentional_stance - the astute reader will note that I skipped the Design Stance. If we truly understood how NNs actually implement all their cognitive processes then we could perhaps apply this to them; if we actually crafted and designed every parameter of its mind. But we are talking about why dice are different.
Great comment. I didn't know about this! Thanks
The latest episode of On The Media also uses this framing.
> On the Media: How Extinction Entered the AI Debate
https://www.wnycstudios.org/podcasts/otm
That's precisely the point: you may be able to understand dice statistically and over the course of long rolls of dice you can extract some properties of the dice. But you won't ever understand any particular roll of the dice.
But importantly for dice we do understand the overarching principles that give rise to this. And dice don't output coherent sentences. Meanwhile in LLM land the analogous "roll of the dice" can result in a coherent response in natural language.
If you use a loaded dice, you can be pretty confident about where it will lands. It may not be 100% accurate, but can be quite close to certain. Without training the weight are pure noises. After training, it leans towards coherent sentences and particular statements.
Yes, and I believe my point still stands. We thoroughly understand the principle by which a loaded die can be intentionally biased despite not being able to predict the outcome of any given throw due to the system in question being a chaotic one.
In contrast, we do not understand LLMs in the same way (nor biological brains). Claiming that anything of that nature is simply biased towards coherent output seems entirely reductive to me - the question is how such coherence arises in the first place. There is no meaning encoded or computation performed by the particular pathway a die travels through the chaotic landscape.
Sure an argument can be made that it's "just" a next token predictor thus how is it really any different from a markov model? Yet the output is not even remotely the same.
> In contrast, we do not understand LLMs in the same way
From my point of view, (not a ML researcher), it’s due to the magic of numbers. The same thing happens with computer vision and neural networks. There’s a bunch of magic weights that get created which has no meaning by themselves, but computing them does help with detecting objects.
So if you take words, derives them into tokens, use the attention techniques to extract the “coherency” aspect, it’s no wonder you can replicate “coherency”. Add reinforcement learning to that to increase towards certain aspects like correct code syntax and you have heavily loaded the dice again.
We have used maths to model chemistry, biology, and physics, as well as economics and sociologic phenomena. Then we use maths (more specifically logic and set theory) to usher in the age of information and computing. Now you want us to act surprised that maths, through ML, can model language.
Maybe further down the line, we can have a simpler set of formulas for language coherency, but for now we have to make to with using the whole internet and a bazillion watts of power to guess the weights for the generic ML model.
I'm comparing it with chess. Chess is pretty complex, complex enough that only a small subset of humans can play it at a very high level. Introducing computers to chess first led to a statistical and brute force approach. But once that had paid off and the results were in people spent a lot of time analyzing those results and this led to an entirely new class of engine that was far more efficient than what had gone before and which performed even better than the 'big iron'.
I would not be surprised at all if we will find that AI will go the same route. The fact that we don't know how it works is where the opportunity for improvement lies.
If nobody knows exactly how, then "at random" sounds about right and the results should be treated as such.
That is, in this case, it should not be used to influence decisions that can start a war.
This is a false dichotomy. We can make statements about the distribution, and we have fuzzy models about certain sets of inputs and outputs. We can steer the outputs.
It’s not completely random. We just don’t understand why the tricks we learned work.
(Fully agree with the second point FWIW)
I agree wholeheartedly about your second sentence, but
“we made this artifact and don’t know why the thing it does looks spookily like cognition”
and
“this artifact makes decisions at random”
are obviously distinct categories and pretending otherwise is silly.
We know why it looks like cognition. Because OpenAI and Antropic put a lot of effort and training to humanize the output and make it sound like a person.
Regardless of negative consequences it brings. They have that project of creating tech god which will save the unborn people thousands years in the future ... so people living now dont matter.
That is why.
“Putting a lot of effort and training” into a dog or an inanimate carbon rod would never result in something that can plausibly substitute for human mental labor and looks likely to eventually surpass us at many tasks, no matter how much you put in.
So I don’t think “labs worked hard” is the same thing is “we know scientifically how these things work in any real level of detail”. The ability to build a thing, even if building it is hard, is not the same thing as understanding of what the thing is or how it works, not even a little bit.
People will just roll their eyes at you and say "the human mind is nothing but a dice roll too" and call you a slope-headed neanderthal before continuing apace.
Try "Emitted".
That's what it did, with no analogy needed.
(But, to be the devil's advocate: the fake can be said about the output of anyone participating here.)
They do make decisions: we present them with options and they decide
We might override them or ignore them or whatever, but they make decisions as much as anybody else or anything else does
The thing is a text generator. It generates text. You can couple that with any code that gives rhe ikkusion of a normal decision workflow, but it does not make any decision more than a software like latex. According to your definition, the latter would “decide” the amount of words to put on a sheet of paper.
Sure they do! Where are you confused?
Can you show me where a human or a dog makes decisions
... neither do you.
>> an LLM is completely opaque
And despite that, although they are not like that in practice as there are too many uncontrolled variables, with temperature at zero, for the same input they produce always the same reply.
If we digitize your brain and reload the checkpoint before you posted, you’ll type this message again in exactly the same way.
Just because a black-box system is deterministic, doesn’t mean it’s understood.
You couldn’t predict the output the first time around, is the point.
Ha, they sure don’t.
They do. Just train your own LLM, not that difficult, and you will have a more controlled environment and you will see they do.
> with temperature at zero, for the same input they produce always the same reply.
Nonsense.
https://thinkingmachines.ai/blog/defeating-nondeterminism-in...
BS. Run them sequentially on a single core, and without fancy speedups enabled, and they do. The algorithm is determinstic. Any non-determinism present with 0 temperature it's not some mysterious LLM-inherent property, but something that can be seen in any large program taking advantage of multi-core, floating point, and other CPU-based parallelism optimization.
Exactly, this is correct.
People often assume they are not because they can ask the same query to the same model and get differences in output, but wrongly conclude that this is some inherent LLM trait, instead of non-determinism added on top of it because of implementational choices that were made.
> Any non-determinism present with 0 temperature it's not some mysterious LLM-inherent property,
Of course it's not mysterious. It is well understood by all who are aware of the fundamental unreliability of all major LLMs in general use today.
The phrasing still makes it sound like it's due to the LLM. It's not. It's do to compiler and CPU, and OS optimizations, and a non-LLM program could suffer the same just as well.
This unreliability is entirely due to the implentation of the LLM. Yes, any other program implemented equally carelessly could suffer the same. But you'd be hard pressed to find any as unreliable as a typical LLM.
Wikipedia is also just a bunch of numbers, so many that you can't memorize them, you may use the same argument to say we don't know why some page links to another.
If you then put Wikipedia, LLM and Brain on a scale to how well they can be understood, you will see that one of them is not like the others.
What about Wikipedia do you think we do not understand? I would say, mechanistically, we can read the code and explain exactly why it does what it does. That doesn’t apply at all to the other two.
That doesn't really matter though, and it just makes the argument stronger.
This is a technology with an inherent tendency of making up false information AND we don't even understand how or why.
That's enough not to entrust these sytems with critical decisions that could start a war.
thats no different from not understanding why a sufficiently complex and obfuscated binary of a program "makes decisions"
We understand how networks compute decisions, though explaining every internal influence remains difficult.
Yeah it speaks poorly of HN that they upvoted this confident nonsense.
Isn’t it convenient that nobody understands? How could we possibly regulate something that isn’t understood? It’s like social media all over again. We can’t be responsible for someone else’s content; it’s not us so you can’t penalize us!!
> Isn’t it convenient that nobody understands? How could we possibly regulate something that isn’t understood?
You mean like the human body? The brain?
If LLMs are like humans, then OpenAI and Anthropic are like slave traders.
I disagree with you that OpenAI and Anthropic are slave traders.
But I think we agree that there are regulated industries built around systems which we don’t fully understand.
My point is that they are using a similar playbook to avoid taking responsibility.
Who is not taking responsibility? You think that anaylst that copy pasted AI slob of such a critical information will be rewarded?
It just feels like the companies behind AIs are spinning their own poor monitoring and criminal (digital) trespassing into something they can't be held responsible for. Even though they are, of course. In the hugging face incident, Anthropic should pay for damages and a fine for malicious hacking. In this incident, whoever signed off on using the tool, and whoever said the intel was good should both be prosecuted or at the very least reprimanded (whatever the rules for a bad interpretation of intel is) and the tool put on hold.
“Convenient” sure sounds like trying to allude to a conspiracy theory. Is that what you’re doing? Why not state your claims or questions directly?