I don't understand how an LLM is able to reason about those things
LLM use language, but it can't "think" about biochemistry
I saw that LLM have reasoning capabilities, which is different from machine learning, but I don't understand how it works.
I don't understand how an LLM is able to reason about those things
LLM use language, but it can't "think" about biochemistry
I saw that LLM have reasoning capabilities, which is different from machine learning, but I don't understand how it works.
An interesting talk I heard at a conference once, that I can neither remember the speaker for or speak to their legitimacy, suggested that we might have some lower form of intelligence encoded into our language. They posed the idea that we have enough unique words, and combination of words, that it starts to have reason unto itself similar to how our neurons and their connection breed intelligence. The idea was that we as humans have baked intelligence into our own speech patterns. It seemed a little to abstract for me, but potentially goes a little way to explaining how a statistical averaging algorithm with some randomness, at scale, starts to look like it very occasionally has a genuinely novel thought.
In The Ticket That Exploded, William S. Burroughs proposes language is a virus in itself, coming from the Outside, and infecting the host with it's control logic. In Radio Free Abemuth, Philip K. Dick attributes a similar possession to a benevolent force, akin to the divine Logos flourishing intelligent development. Both seem open to an impersonal agency that maps to intelligent systems encoded in their transfer protocols.
The english language pattern is definitely shaping our thoughts and limiting our ideas. Just the whole idea of going from some abstract thought to actually making it out in tangible language, I mean no matter the thought it's a lossy transfer into a medium that lacks all the dimensionality of subconscious thought, neurotransmitter action, sensory information, and physiological response.
There's also something to consider with lower level vs higher level abstractions in language. E.g. jargon. One short word could have a 200 page thesis behind it defining all the ramifications. Talk about compression of information.
Now imagine if our language lacked say the mechanism of jargon, of using some meta word to define thousands of stringed together words at once. Every idea like "car" would have to be described from first principles. The species would probably never develop technology with this sort of language pattern present. If we could somehow level up beyond our current abstraction level, maybe that would make us even smarter, able to handle bigger ideas quicker in real time.
Even more simply than all this: I can only speak about what I have english words for.
If you are fluent in a second language, you understand that the only way to become truly fluent is not just to learn the tangible aspects - the vocabulary, grammar, references, expressions, etc - you have to learn to think like the language/culture.
It’s a combination of cultural assumptions, facial expressions and affectations, thinking patterns, and a whole cultural upbringing that can lead you to very different mental processes and natural conclusions starting from the same words and phrases.
Language absolutely encodes a certain form of intelligence. A lot of those things are reflected not just in the totality of the culture but the language itself. Being fluent leads you to different thinking patterns and different conclusions when processing in that language.
At one point there was a universal language, and later either recreated or scrambled, allowing for humans to have many languages for increased confusion.
Now that we are once again attempting to unify our language we find ourselves in a pursuit to build something to escape the Earth.
There is no evidence there was ever a universal language. Or am I missing a joke here?
Sage thoughts, trash88
https://inv.nadeko.net/watch?v=Or_3tlEOLj4&pp=ugUEEgJlbg%3D%...
[dead]
I thought that was how most people understood LLM’s capabilities? We have spent millenia creating language to map onto our world. Therefore, implicit in that language is a simulacrum of our world.
An interesting theory that would cast some of the more interestingly phrased verses of the Bible in a new light, e.g.
“In the beginning was the Word, and the Word was with God, and the Word was God.” - John 1:1
I feel like I can feel this happening in my mind in real time. Something like: the part of my brain that thinks thoughts is fairly rudimentary, basically just impressions or hunches--but then there's another part which translates them into words and grammar, and when it takes an impression it can translate it into something fairly sophisticated and intelligent, because it's somehow necessary in order to create a sentence which actually captures the impression.
Does that mean the language(s) we speak determine how intelligent we are? Could learning French, for example—often considered a more expressive language—make a native English speaker more intelligent or even more compassionate?
It's arguable; see the Sapir-Worf hypothesis in the strong and weak forms.
I would argue the languages with the most colors belong to the most intelligent.
Does that help explain why learning a word for something can help understand the concept of it?
Isn't it more to do with being able to describe truth with language
Very intriguing, brings to mind Sapir-Whorf a little bit. You wouldn't happen to remember the name of the speaker or the conference, would you?
Does that explain why different countries that speak different languages have different engineering cultures? Like is german better suited towards engineering than english for example?
Michael Levin, the Biologist.
https://inv.nadeko.net/watch?v=Or_3tlEOLj4&pp=ugUEEgJlbg%3D%...
Interesting. Would this apply to any rich enough system of expression, like music or art? Or is there something specific about language that makes it different?
It's pretty clear reading from these comments that most HN members have a 2023-era impression of LLMs.
Modern chain-of-thought models with RL post training on verifiable tasks + realistic environments + rubrics are worlds apart from models trained on a simple next token prediction objective.
More money goes into the rubrics and RL environments than individual training runs themselves.
(Yes, at inference-time LLMs still output words one at a time, much like human speakers. But don't confuse the mechanism with the training objective.)
[dead]
But but but....I was told it was a stochastic parrot! I liked that idea because it appealed to my vanity, and it described the gibberish produced by older models with bad prompting, and that was enough for me thank you.
/s
Nobody knows how it works, really. It just turned out that if you try to predict the next word then you get intelligent behavior, depending on amount of training data, and the size and topology of the network. But again, nobody knows why, and what the limits are.
Agreed. We went this direction for our golems, djinns, and other mechanistic minds because we believe it sort of reflects the primitives of our own neurons (which we also don't fully grok).
Linus Torvalds:
~"Predicting the next token is not an insult. It's pretty much what we all do."
That’s my and probably most people’s understanding.
I have a feeling we know more than that about how it works.
We don't know how our own intelligence works, much less anything else's
We built LLMs.
We didn't build our brain.
Typically when you build something you have a decent idea how it works.
It's more accurate to say we grew them. That is the breakthrough of Deep Learning. We left the hard part to the machine (learning how to do what you want it to do) to figure out during training.
And that means we are not privy to whatever things it has learnt in its trillions of weights.
https://xkcd.com/1838/
We might have built them, we sure as hell didn't design them. And no, we do not have a decent idea about how it works.
I heard someone who studies this sort of thing say basically what biological neurons are trying to do is predict as well. Predicting what exactly? I’m not sure. The next time they should fire or something. I can’t find the YouTube video now.
In the last few decades, there has been an increased interest in the role of prediction in language comprehension. The idea that people predict (i.e., context-based pre-activation of upcoming linguistic input) was deemed controversial at first. However, present-day theories of language comprehension have embraced linguistic prediction as the main reason why language processing tends to be so effortless, accurate, and efficient.
https://www.earth.com/news/our-brains-are-constantly-working...
https://www.psycholinguistics.com/gerry_altmann/research/pap...
https://www.tandfonline.com/doi/pdf/10.1080/23273798.2020.18...
https://onlinelibrary.wiley.com/doi/10.1111/j.1551-6709.2009...
Interesting.
And I also found the video I was referring to https://www.youtube.com/watch?v=FHQfmJEpRmU
Predict activations in adjacent neurons, roughly; the relevant keyword is "Hebbian learning" (and "predictive coding" at a higher level).
Predicting reality, under the "controlled hallucination" framing, corrected by sensory error signals. The brain has no access to ground truth, only input data that helps correct the hallucination.
Based on lots of human interactions, I think there are a lot of human beings out there who mentally aren’t much more than “next word predictors” who happen to be made of meat+neurons instead of silicon+code.
The cure for HER2- metastatic breast cancer is a simple matter of ...
Please predict the next word.
Intelligence is implicit in language understanding. The best possible next-word-predictor is omniscient.
> The best possible next-word-predictor is omniscient.
Omniscient for the set of "meaning" embedded into it's training set. It's not broadly omniscient, big difference.
Sure, but same for any intelligence. The "training set" is just "the environment of adaptation".
Careful, you're dangerously close to an argument for the necessity of embodiment /s
Shooting from the hip: Embodiment is simply a highly sophisticated harness.
Right, but what's the limit of what you can deduce computationally from truly vast training sets? How much structure is there in the subtext of what's written down? It looks like there's rather a lot.
The cure for HER2- metastatic breast cancer is a simple matter of [intensive well-funded research]
That wasn't too hard, maybe I'm superintelligent?
What does omniscience have to do with reasoning? If you know everything, you don’t have to reason. But these next-word-predictors aren’t omniscient.
Omniscience doesn’t imply that you have to store all the information, but that you can retrieve/reconstruct it, and reasoning allows it. In fact would be impossible for any physical intelligence to store all the information as plain as it is infinite.
Intelligence as a measure of the ability to define predictive models of certain problems (and their solutions).
Promoting LLMs is encoding the problem we want into the query vectors, and through the magic of the complex training and the power of operations in a very large dimensional abstract space the AI can manipulate the representations, and iteratively approximate solutions. (And using bigger and bigger contexts and better encodings it can form better models.)
Here's my grok of it: Deep learning models progressively abstract a concept presented at the input by passing the input through many sequential layers () until an output layer transforms the output of the final layer into something interpretable, such as an indication of what token to predict next, or a classification, or whatever. The transformer architecture futhermore offers layers that allow different parts of the previous layer's output to sort of mix with each other in complex ways. As you get into greater levels of abstraction, the attention process is mixing very abstract concepts with each other in a nonetheless highly structured manner. I believe this is where the intelligence lives.
sometimes with residual connections, but we can ignore that for sake of simplicity.
Language emanates from intelligence. That means the patterns and structure that make up human intelligence will appear in language. LLMs are created through so much language training that they can approximate (and now to some degree exceed) human intelligence using pattern recognition, statistics, and autocomplete (in layman’s terms).
Not sure how it is now, but early “reasoning” was simply the big labs sticking “wait a minute, what if I…” type language blocks into the process to trigger something like our own internal reasoning.
Compression and understanding are correlated.
Biochemistry is abstract to us humans too, we can only create hypotheses, and validate them experimentally.
And for those hypotheses - we use language.
Most human reasoning happens within language - even mathematics is an abstraction that allows us to map concepts we don’t natively hold into a linguistic processing layer.
How do you think? I think with words.
Theres more than words in our minds. Think harder are you absolutely sure? You REASON with words but your ideas dont form just from you reasoning. The ideas just seem to come out of nowhere to the part of your brain that then reasons around them.
Do you have an internal monologue?
I don't. I seem to think at a more abstract, pre-verbal level rather than through an internal voice.
Some studies suggest that frequent internal monologue may occur in roughly 30–50% of people [1], but the research is based on relatively small samples.
[1] https://www.psychologytoday.com/us/blog/intersections/202304...
If you try and think before you speak, does it just not work?
I have different modes depending on what I'm doing, but I think usually I'm reasoning non-verbally
I tend to agree with Albert Einstein below; there's a very physical/spatial aspect to my problem solving before it can be translated to words. I work in software so there's nothing innately physical about it. Never put much thought to it until LLMs brought it up for debate.
"The words of the language, as they are written or spoken, do not seem to play any role in my mechanism of thought. The psychical entities which seem to serve as elements in thought are certain signs and more or less clear images which can be "voluntarily" reproduced and combined....From a psychological viewpoint this combinatory play seems to be the essential feature in productive thought....The...elements are, in my case, of visual and some of muscular type. Conventional words or other signs have to be sought for laboriously only in a secondary stage, when the mentioned associative play is sufficiently established and can be reproduced at will."
How do you know that it's the words driving the thinking, rather than the stream of words just being an observable trace tacked onto the actual thinking?
These days, words. When I was in an environment where language swapping between 4 to 5 languages was common, I thought in pictures and described it in the correct language for the audience. It was a plasticity mind trip.
Also saved pesos on the charge-per-text SMS schemes the local phone companies used because we could embed information across so many options.
You think with and without words. When you have to pee, it isn't like you speak to yourself "Gee, pinch in the loins, I guess that must mean must have to pee. Alright legs, get me up off my butt. Left right left right left right. Stop. Hand, get the zipper going. Johnson, your turn now."
Nope. You up and pee.
My friend has aphantasia and cannot think with words, sounds, or pictures.
How do they describe how they think?
Even easier, what are they doing when they are reading?
You don't need to imagine words, sounds, or pictures to absorb information. There's the live image of the text, but you don't need visualization.
Ha ha, you completely understand how you work though.
I don't think LLMs currently have direct reasoning abilities, but as we make them more complicated (MoE, RL) I think we're getting better at learning an implicit world model that guides the token output distribution towards making good hypotheses.
AI is way beyond conventional LLM architecture now. It combines LLMs with search + RL. The traditional LLM architecture hit a wall around GPT-4o. Arc AGI evals show this.
All that extra is clear as day compared to the mystery of how neural network training decides to divide and balance the weights in even small neutral networks.
We can, at best, approach a good set of weights, even in tiny neural networks.
Imagine if we found a way to calculate the exact optimal weights for a given loss function. I mean, there is an exact optimal solution, it exists, but we can't find it exactly, even for a neural network with just 50 parameters.
There is no point in that because the loss function itself is already an approximation. No one knows what is the exact loss function for any given non-trivial real-world task.
I mean, things humans defined can be pretty clear. Like your electricity rate. Natural systems less so. Not pretending no complexity in human made things, but at least some models can be fully specified.
Granted, I feel like munging gigabytes of text data (i.e. G, A, T and Cs) would be something LLMs would be good at
I imagine it's writing a story about a character doing those things and then reading the story and acting on it.
Stephen Wolfram had a great description of this effect in the early days (GPT 3.5 era):
Machine learning trains the network to do... anything that you reward it for. If you keep training, it keeps getting better.
Next word prediction can always keep getting better.
At first, simply "learning" spelling is what makes the predictions better because tokens are word chunks, not always whole words.
Then, the models "run out of steam" and can't get any better by learning more spelling rules, but the gradient descent forces them to get better... so they do... by learning the rules of grammar.
At this point the AIs can output correctly spelled and grammatically coherent sentences, but the sentences ramble on about nonsense topics.
So what happens next as the models run out of grammar rules is that they're forced to learn the rules "above grammar": logic, world knowledge, coherent story telling, etc.
At some point they learn to output pages and pages of fluid, coherent text, but... if they're not smart, if they don't think, and if they don't know what they're talking about, then they're still "suboptimal" and their forced gradient descent will make them close those gaps.
Eventually, the only way they can improve at "next token prediction" is by building up to human-like intelligence, including an inner monologue, theory of mind, and everything.
We can even read their "thoughts": https://transformer-circuits.pub/2026/workspace/index.html
What's to say it "can't" think? These are not tasks you can do without thinking.
I don't see why its that crazy that a system with a huge amount of parameters starts to exhibit emergent behavior
I don't think anyone knows, not even the LLMs.
I mean, the subtlety of the neural network weights that emerge from training are not fully comprehended by anyone, man or machine.
Every individual calculation is understood, and every step of training is understood, but the exact nature of those weights that divide the responsibility of responding to subtle changes of input in intelligent ways is beyond me.
No worries, no one does. Exactly like no one knows how the brain reasons either.
It's really good at pattern recognition.
So I'm not sure how it knows to be 'surprised' that alone is pretty fascinating.
If I were to guess, being pleasantly surprised is just a learned appropriate social response from the expectation of receiving a reward and as such, that social norm is codified sufficiently enough in our writings that it appears in LLMs output.
It’s sort of like all the people who will ask Claude or GPT to validate their complete nonsense and receive unyielding praise for it, the models just learned that this is the best received response based on training data and RL.
I bet these same sorts of expressions can be found in practically every failed attempt as well.
[dead]