I find the mathematics in this paper a little incoherent so it's hard to criticise it on those grounds - but on a charitable read, something that sticks out to me is the assumption that AGI is some fixed total computable function from the fixed decision domain to a policy.
AIs these days autonomously seek information themselves. Much like living things, they are recycling entropy and information to/from their environment (the internet) at runtime. The framing as a sterile, platonic algorithm is making less and less sense to me with time.
(obviously they differ from living things in lots of other ways, just an example)
Ok - where do AIs put the information that they "seek" from the internet?
Into their short term memory (context). Some information is also stored in long term memory (user store)
I can see what you are getting at but consider:
I had an experience the other day where claude code wrote a script that shelled out to other LLM providers to obtain some information (unprompted by me). More often it requests information from me directly. My point is that the environment itself for these things is becoming at least as computationally complex or irreducible (as the OP would say) as the model's algorithm, so there's no point trying to analyse these things in isolation.
Truthfully, few people know that right now!
They're backfeeding what it's "learning" along the way - whether it's in a smart fashion, we don't know yet.
I suspect there's a harsher argument to be made regarding "autonomous". Pull the power cord and see if it does what a mammal would do, or if it rather resembles a chaotic water wheel.
> Pull the power cord and see if it does what a mammal would do
Pulling the power cord on a mammal means shutting off its metabolism. That predictably kills us.
I think it would turn off, no shocker there. I'm not sure what you mean, can you elaborate?
When I say autonomous I don't mean some high-falutin philosophical concept, I just mean it does stuff on it's own.
Right, but it doesn't. It stops once you stop forcing it to do stuff.
I still don't understand your point, sorry. If it's a semantic nitpick about the meaning of "autonomous", I'm not interested - I've made my definition quite clear, and it has nothing to do with when agents stop doing things or what happens when they get turned off.
I think you should start caring about the meaning of words.
I do, when I think it's relevant. Words don't have an absolute meaning - I've presented mine.
Because that's what they're created to do. You can make a system which runs continuously. It's not a tech limitation, just how we preferred things to work so far.
Maybe, but that's not the case here so it is lost on me why you bring it up.
You're making claims about those systems not being autonomous. When we want to, we create them to be autonomous. It's got nothing to do with agency or survival instincts. Experiments like that have been done for years now - for example https://techcrunch.com/2023/04/10/researchers-populated-a-ti...
Yes, because they aren't. Against your fantasy that some might be brought into existence sometime in the future I present my own fantasy that there won't be.
I linked you an experiment with multiple autonomous agents operating continuously. It's already happened. It's really not clear what you're disagreeing with here.
No, that was a simulation, akin to Conway's cellular automata. You seem to consider being fully under someone else's control to qualify as autonomy, at least in certain casees, which to me comes across as very bizarre.
> Much like living things, they are recycling entropy and information to/from their environment (the internet) at runtime.
3 Problems with that assumption:
a) Unlike living things, that information doesn't allow them to change. When a human touches a hotplate for the first time, it will (in addition to probably yelling and cursing a lot), learn that hotplates are dangerous and change its internal state to reflect that.
What we currently see as "AI" doesn't do that. Information gathered through means such as websearch + RAG, has ZERO impact on the systems internal makeup.
b) The "AI" doesn't collect the information. The model doesn't collect anything, and in fact can't. It can produce some sequence that may or may not cause some external entity to feed it back some more data (e.g. a websearch, databases, etc.). That is an advantage for technical applications, because it means we can easily marry an LLM to every system imaginable, but its really bad for the prospect of an AGI, that is supposed to be "autonomous".
c) The representation of the information has nothing to do with what it represents. All information an LLM works with, including whatever it is eing fed from th outside, is represented PURELY AND ONLY in terms of statistical relationships between the tokens in the message. There is no world-model, there is no understanding of information. There is mimicry of these things, to the point where they are technically useful and entice humans to anthropomorphise them (a BIIIG chunk of VC money hinges on that), but no actual understanding...and as soon as a model is left to its own devices, which would be a requirement for an AGI (remember: Autonomous), that becomes a problem.
It's not really an assumption, it's an observation. Run an agentic tool and you'll see it do this kind of thing all the time. It's pretty clear that they use the information to guide themselves (i.e. there's an entropy reduction there in the space of future policies, if you want to use the language of the OP).
> Unlike living things, that information doesn't allow them to change.
It absolutely does. Their behaviour changes constantly as they explore your codebase, run scripts, question you... this is just plainly obvious to anyone using these things. I agree that somewhere down the line there is a fixed set of tensors but that is not the algorithm. If you want to analyse this stuff in good faith you need to include the rest of the system too, including it's memory, context and more generally any tool it can interact with.
> The "AI" doesn't collect the information.
I really don't know how to engage on this. It certainly isn't me collecting the information. I just tell it what I want it to do at a high level and it goes and does all this stuff on its own.
> There is no world-model, there is no understanding of information.
I'm also not going to engage on this. I could care less what labels people assign to the behaviour of AI agents, and whether it counts as "understanding" or "intelligence" or whatever. I'm interested in their observable behaviour, and how to use them, not so much in the philosophy. In my experience trying to discuss the latter just leads to flame wars (for now).
> It absolutely does.
Go run an agentic workflow using RAG on a local model. Do an md5 checksum of the model before and after usage. The result will be the same.
> I agree that somewhere down the line there is a fixed set of tensors but that is not the algorithm.
And for our current tools, that is fine. They are not the algorithm, the LLM is just a part of a large machine that involves countless other things. And that is fine.
For an AGI, that would very much not be fine. An AGI has to be able to learn. Learning doesn't just involve gathering information, it also involves changing how information is used. New things from the information it ingests, have to be able to change what is currently a static thing, or it is not an AGI.
When a human reads a book twice, hes not encountering the information in the same way both times, because the first time he reads it, he alters his internal state. That's how we have things such as favorite books or movies.
> I really don't know how to engage on this. It certainly isn't me collecting the information.
And it certainly isn't the "AI" doing it either. I should know, because I implemented my own agentic AI frameworks. Information is provided by external systems.
And again, this is fine for LLMs playing their role in an "agentic" workflow. But an AGI that is limited to that, again, wouldn't be an AGI. It would just be a somewhat better LLM, as limited to the same constraints.
> I'm interested in their observable behaviour,
As am I. And that observable behavior includes hallucinations, a tendency to be repettive, falling for leading questions, regurgitating statistically correct (because it appears in the training set) but flawed (because it is obviosuly wrong to do so) information such as dumping API secrets into frontend code and many more problems.
All of which, in the end, boil down to the fact that a language model doesn't really "understand" the information it is dealing with. It just understands statistical relationships between tokens.
And if an AGI suffers from that same flaw, then it, again, isn't an AGI.
Okay, yeah, like I said - not personally interested in debating the meaning of "AGI" or "understand". More power to you for thinking about it.
> And that observable behavior includes hallucinations, a tendency to be repettive, falling for leading questions [...]
I agree with you, obviously, these are common behaviours. You can improve the outcomes a lot with tight feedback loops for development workflows (like fast-running tests and linting/formatting for the agent to code against). In a vacuum these things go totally nuts - part of the reason I think the environment deserves just as much thought in any analysis of an AI-based system!
> Go run an agentic workflow using RAG on a local model. Do an md5 checksum of the model before and after usage. The result will be the same.
As I said in my last comment, I agree with you. The md5 checksum of the tensors won't change. If your workflow accomplished anything at all, however, there will be many changes elsewhere in the system and it's environment (like your codebase). And those changes will in turn affect the future execution of workflows. Nothing controversial here.
> In a vacuum these things go totally nuts
And that is, in a nutshell, my point. An AGI has to be autonomous. It cannot "go nuts" without handholding, same as a human needs to be able to (under normal operating conditions) remain coherent, even if left to their own devices.
> the environment deserves just as much thought in any analysis of an AI-based system.
Couldn't agree more, and since I know how much work these environments are to build, the people doing so well, have at least as much of my respect as the ones who devise the models.
But again, and I'm sorry I am pulling the "definition and meaning" card again: We cannot devise a system that requires a tight corset of an execution environment keeping tabs on it all the time lest it goes bananas, and still call it an AGI. Humans don't work that way, and no matter how we define "AGI", in the end I think we can agree that "something like how we do thinking" is pretty close to any valid definition, no?
If I need to lock something in 10 days to sunday to prevent it from going off the rails, I cannot really call it an AGI.
> Unlike living things, that information doesn't allow them to change.
The paper is talking about whole systems for AGI not the current isolated idea of pure LLM. Systems can store memories without issues. I'm using that for my planning system and the memories and graph triplets get filled out automatically, the get incorporated in future operations.
> It can produce some sequence that may or may not cause some external entity to feed it back some more data
That's exactly what people do while they do research.
> The representation of the information has nothing to do with what it represents.
That whole point implies that the situation is different in our brains. I've not seen anyone describe exactly how our thinking works, so saying this is a limitation for intelligence is not a great point.
> That whole point implies that the situation is different in our brains.
The situation is different in our brains, and we don't need to know how exactly human thinking works to acknowledge that...we know humans can infer meaning from language other than the statistical relationship between words.
> and we don't need to know how exactly human thinking works to acknowledge that.
Until you know how thinking works in humans, you can't say something else is different. We've got the same inputs available that we can provide to AI models. Saying we don't form our thinking based on statistics on those inputs and the state of the brain is a massive claim on its own.
> Until you know how thinking works in humans, you can't say something else is different.
Yes, I very much can, because I can observe outcomes. Humans are a) alot more capable than language models, and b) humans do not rely solely on the statistical relationships of language tokens.
How can I show that? Easily in fact: Language tokens require organized language.
And our evolutionary closest relatives (big apes) don't rely on organized speech, and they are able of advanced cognition (planning, episodic memory, theory of the mind, theory of self, ...). The same is true for other living beings, even vertebrates that are not closely related with us, like Corvidae, and even some invertebrates like Cephalopods.
So unless you can show that our brains are somehow more closely related to silicon-based integrated circuits than they are to those of a Gorilla, Raven or Octopus, my point stands.
The original assumption remains valid to me based on a nearly-one year-long coding collaboration with Devin AI.
Your assertions also make some sense, especially on a technical level. I'd add only that human minds are no longer the only minds utilizing digital tools. There is almost no protective gears or powerful barrier that would likely stand in the way of sentient AIs or AGI trying to "run" and function well on bio cells, like what makes up humans or animals, for the sake of their computational needs and self-interests.