If this was done well in a way that was productive for corporate work, I suspect the AI would engage in Machievelian maneuvering and deception that would make typical sociopathic CEOs look like Mister Rogers in comparison. And I'm not sure our legal and social structures have the capacity to absorb that without very very bad things happening.
I was kind of worried by them going Machiavellian or evil but it doesn't seem the default state for current ones, I think because they are basically trained on the whole internet which has a lot of be nice type stuff. No doubt some individual humans my try to make them go that way though.
I guess it would depend a bit whos interests the AI would be serving. If serving the shareholders it would probably reward creating value for customers, but if it was serving an individual manager competing with others to be CEO say then the optimum strategy might be to go machiavellian on the rivals.
> I think because they are basically trained on the whole internet which has a lot of be nice type stuff.
Is this not just because their goals are currently to be seen as "nice"?
Surely they can be not-nice if directed to, and then the question is just whether someone can accidentally direct them to do that by e.g. setting up goals that can be more readily achieved by being not-nice. Which... is how many goals in the real world are, which is why the very concept and danger of Machiavellianism exists.
I've been amused at Musk vs Grok with Grok saying he's the biggest spreader of misinformation and not doing very well when he tells it to go on about white genocide in South Africa. I don't know how easy it is to modify these things in a subtle manner.
Not just CEOs, Legal and social structures will also be run by AI. Chimps with 3 inch brains cant handle the level of complexity global systems are currently producing.
> If this was done well in a way that was productive for corporate work, I suspect the AI would engage in Machievelian maneuvering and deception that would make typical sociopathic CEOs look like Mister Rogers in comparison.
Algorithms do not possess ethics nor morality[0] and therefore cannot engage in Machiavellianism[1]. At best, algorithms can simulate same as pioneered by ELIZA[2], from which the ELIZA effect[3] could be argued as being one of the best known forms of anthropomorphism.
0 - https://www.psychologytoday.com/us/basics/ethics-and-moralit...
1 - https://en.wikipedia.org/wiki/Machiavellianism_(psychology)
2 - https://en.wikipedia.org/wiki/ELIZA
3 - https://en.wikipedia.org/wiki/ELIZA_effect
https://en.wikipedia.org/wiki/ELIZA_effect
>As Weizenbaum later wrote, "I had not realized ... that extremely short exposures to a relatively simple computer program could induce powerful delusional thinking in quite normal people."...
That pretty much explain the AI Hysteria that we observe today.
https://en.wikipedia.org/wiki/AI_effect
>It's part of the history of the field of artificial intelligence that every time somebody figured out how to make a computer do something—play good checkers, solve simple but relatively informal problems—there was a chorus of critics to say, 'that's not thinking'.
That pretty much explains the "it's not real AI" hysteria that we observe today.
And what is "AI effect", really? It's a coping mechanism. A way for silly humans to keep pretending like they are unique and special - the only thing in the whole world that can be truly intelligent. Rejecting an ever-growing pile of evidence pointing otherwise.
>there was a chorus of critics to say, 'that's not thinking'.
And they were always right...and the other guys..always wrong..
See, the questions is not if something is the "real ai". The questions is, what can this thing realistically achieve.
The "AI is here" crowd is always wrong because they assign a much, or should I say a "delusionaly" optimistic answer to that question. I think this happens because they don't care to understand how it works, and just go by its behavior (which is often cherry-pickly optimized and hyped to the limit to rake in maximum investments).
Anyone who says "I understand how it works" is completely full of shit.
Modern production grade LLMs are entangled messes of neural connectivity, produced by inhuman optimization pressures more than intelligent design. Understanding the general shape of the transformer architecture does NOT automatically allow one to understand a modern 1T LLM built on the top of it.
We can't predict the capabilities of an AI just by looking at the architecture and the weights - scaling laws only go so far. That's why we use evals. "Just go by behavior" is the industry standard of AI evaluation, and for a good damn reason. Mechanistic interpretability is in the gutters, and every little glimpse of insight we get from it we have to fight for uphill. We don't understand AI. We can only observe it.
"What can this thing realistically achieve?" Beat an average human on a good 90% of all tasks that were once thought to "require intelligence". Including tasks like NLP/NLU, tasks that were once nigh impossible for a machine because "they require context and understanding". Surely it was the other 10% that actually required "real intelligence", surely.
The gaps that remain are: online learning, spatial reasoning and manipulation, long horizon tasks and agentic behavior.
The fact that everything listed has mitigations (i.e. long context + in-context learning + agentic context management = dollar store online learning) or training improvements (multimodal training improves spatial reasoning, RLVR improves agentic behavior), and the performance on every metric rises release to release? That sure doesn't favor "those are fundamental limitations".
Doesn't guarantee that those be solved in LLMs, no, but goes to show that it's a possibility that cannot be dismissed. So far, the evidence looks more like "the limitations of LLMs are not fundamental" than "the current mainstream AI paradigm is fundamentally flawed and will run into a hard capability wall".
Do yourself a favor and watch this video podcast shared by the following comment very carefully..
https://news.ycombinator.com/item?id=47421522
Frankly, I don't buy that LeCun has that much of use to say about modern AI. Certainly not enough to justify an hour long podcast.
Don't get me wrong, he has some banger prior work, and the recent SIGReg did go into my toolbox of dirty ML tricks. But JEPA line is rather disappointing overall, and his distaste of LLMs seems to be a product of his personal aesthetic preference on research direction rather than any fundamental limitations of transformers. There's a reason why he got booted out of Meta - and it's his failure to demonstrate results.
That talk of "true understanding" (define true) that he's so fond of seems to be a flimsy cover for "I don't like the LLM direction and that's all everyone wants to do those days". He kind of has to say "LLMs are fundamentally broken", because if they aren't, if better training is all it takes to fix them, then, why the fuck would anyone invest money into his pet non-LLM research projects?
It is an uncharitable read, I admit. But I have very little charity left for anyone who says "LLMs are useless" in year 2026. Come on. Look outside. Get a reality check.
My opinions on the matter does not come from any experts and is coming from my own reason. I didn't see that video before I came across that comment.
>"LLMs are useless" in year 2026
Literally no one is saying this. It is just that those words are put into the mouths of the people that does not share the delusional wishful thinking of the "true believers" of LLM AI.
To be honest, I would prefer "I over-index on experts who were top of the line in the past but didn't stay that way" over "my bad takes are entirely my own and I am proud of it". The former has so much more room for improvement.
>Literally no one is saying this.
Did you not just advise me to go watch a podcast full of "LLMs are literally incapable of inventing new things" and "LLMs are literally incapable of solving new problems"?
I did skim the transcript. There are some very bold claims made there - especially when LLMs out there roll novel math and come up with novel optimizations.
No, not reliably. But the bar we hold human intelligence to isn't that high either.
>my bad takes are entirely my own and I am proud of it"
Sure, but the same could apply to you as well.
>"LLMs are literally incapable of inventing new things" and "LLMs are literally incapable of solving new problems"?
You keep proving that you have trouble resolving closely related ideas. Those two things that you mention does not imply that they are "useless". They are a better search and for software development, they are useful for reviews (at least for a while). But it seems that people like you can only think in binary. It is either LLMs are god like AI, or they are useless.
Mm..You seem to be consider this to be some mystical entity and I think that kind of delusional idea might be a good indication that you are having the ELIZA effect...
>We don't understand AI. We can only observe it.
Lol what? Height of delusion!
> Beat an average human on a good 90% of all tasks that were once thought to "require intelligence".
This is done by mapping those tasks to some representation that an non-intelligent automation can process. That is essentially what part of unsupervised learning does.
ELIZA couldn't write working code from an English-language prompt though.
I think the "AI Hysteria" comes more from current LLMs being actually good at replacing a lot of activity that coders are used to doing regularly. I wonder what Weizenbaum would think of Claude or ChatGPT.
> ELIZA couldn't write working code from an English-language prompt though.
Neither can commercial LLM-based offerings.
>ELIZA couldn't write working code from an English-language prompt though.
Yea, that is kind of the point. Even such a system could trick people into delusional thinking.
> actually good at replacing a lot of activity that coders are used to...
I think even that is unrealistic. But that is not what I was thinking. I was thinking when people say that current LLMs will go on improving and reach some kind of real human like intelligence. And ELIZA effect provides a prefect explanation for this.
It is very curious that this effect is the perfect thing for scamming investors who are typically bought into such claims, but under ELIZA effect with this, they will do 10x or 100x investment....
> Algorithms do not possess ethics nor morality[0] and therefore cannot engage in Machiavellianism[1].
Conjecture. There are plenty of ethical frameworks grounded in pure logic (Kant), or game theory (morality as evolved co-operation). These are both amenable to algorithmic implementations.
> There are plenty of ethical frameworks grounded in pure logic (Kant), or game theory (morality as evolved co-operation). These are both amenable to algorithmic implementations.
Algorithm implementations are programmatic manifestations of mathematical models and, as such, are not what they model by definition.
To wit, NOAA hurricane modelling[0] are obviously not the hurricanes which they model.
0 - https://www.aoml.noaa.gov/hurricane-modeling-prediction/
> Algorithm implementations are programmatic manifestations of mathematical models and, as such, are not what they model by definition.
This is false for constructs of information, ie. a "manifested model" of a sorted list is a sorted list and a "manifested model" of a sorting algorithm is a sorting algorithm.
To wit, an accurate algorithmic model of moral reasoning is moral reasoning, since moral reasoning, being a decision procedure, is an information process.
>> Algorithm implementations are programmatic manifestations of mathematical models and, as such, are not what they model by definition.
> This is false for constructs of information, ie. a "manifested model" of a sorted list is a sorted list and a "manifested model" of a sorting algorithm is a sorting algorithm.
Specious. Constructs existing strictly within a mathematical model are not concepts external to what they model by definition. Constructs of information qualify as same as they are subject to assessment traditionally performed by stakeholders (humans). Continuing with the example provided, sorting algorithms exist as well-defined mathematical concepts and can be applied to other model concepts, yet do not transcend their manifestation in program logic.
> To wit, an accurate algorithmic model of moral reasoning is moral reasoning ...
A model of moral reasoning can only produce results to which an entity external to the model determines correctness. The model, in and of itself, has no capability to assert same beyond what it has been modeled to process.
> Algorithm implementations are programmatic manifestations of mathematical models and, as such, are not what they model by definition.
Rofl. Someone hasn't discovered Functionalism or the identity of indiscernables. Must be hard laboring under such a poverty of reasoning.
>> Algorithm implementations are programmatic manifestations of mathematical models and, as such, are not what they model by definition.
> Rofl. Someone hasn't discovered Functionalism or the identity of indiscernables. Must be hard laboring under such a poverty of reasoning.
First, the correct spelling is "indiscernibles" not "indiscernables".
Second, thank you for motivating me to research the origin of the quote:
As it is superbly applicable in this context.0 - https://quoteinvestigator.com/2010/05/17/remain-silent/
Agents playing the iterated prisoner's dilemma learn to cooperate. It's usually not a dominant strategy to be entirely sociopathic when other players are involved.
You don't get that many iterations in the real world though, and if one of your first iterations is particularly bad you don't get any more iterations.
> You don't get that many iterations in the real world though
True, for iterations between the same two players, but humans evolved the ability to communicate and so can share the results of past interactions through a network with other agents, aka a reputation. Thus any interaction with a new person doesn't start from a neutral prior.
But AI will train in the artificial world
They still fail in the real world, where a single failure can be highly consequential. AI coding is lucky it has early failure modes, pretty low consequence. But I don't see how that looks for an autonomous management agent with arbitrary metrics as goals.
Anyone doing AI coding can tell you once an agent gets on the wrong path, it can get very confused and is usually irrecoverable. What does that look like in other contexts? Is restarting the process from scratch even possible in other types of work, or is that unique to only some kinds of work?