Norbert Wiener in 1960:
"As is now generally admitted, over a limited range of operation, machines act far more rapidly than human beings and are far more precise in performing the details of their operations. This being the case, even when machines do not in any way transcend man's intelligence, they very well may, and often do, transcend man in the performance of tasks. An intelligent understanding of their mode of performance may be delayed until long after the task which they have been set has been completed. This means that though machines are theoretically subject to human criticism, such criticism may be ineffective until long after it is relevant. To be effective in warding off disastrous consequences, our understanding of our man-made machines should in general develop _pari passu_ with the performance of the machine. By the very slowness of our human actions, our effective control of our machines may be nullified. By the time we are able to react to information conveyed by our senses and stop the car we are driving, it may already have run head on into a wall."
"In neurophysiological language, ataxia can be quite as much of a deprivation as paralysis. A patient with locomotor ataxia may not suffer from any defect of his muscles or motor nerves, but if his muscles and tendons and organs do not tell him exactly what position he is in, and whether the tensions to which his organs are subjected will or will not lead to his falling, he will be unable to stand up. Similarly, when a machine constructed by us is capable of operating on its incoming data at a pace which we cannot keep, we may not know, until too late, when to turn it off."
Source: https://www.cs.umd.edu/users/gasarch/BLOGPAPERS/moral.pdf
What a paper!
And you missed an even MORE relevant excerpt!!
"Complete subservience and complete intelligence do not go together."
I'm not convinced this is true. Perhaps for a human it is, but we can give an artificial mind whatever properties we want.
Even for people, what about e.g. the extremely intelligent military general who is absolutely loyal to his king? (Of course, some generals do lead coups and you can't know in advance which ones, but I'd think there are plenty who have undying loyalty, and I don't think it correlates to overall intelligence!)
>I'm not convinced this is true. Perhaps for a human it is, but we can give an artificial mind whatever properties we want.
Just because it's artificial doesn't mean you can 'give it any properties you want'. We certainly can't do that for Deep ANNs.
>Even for people, what about e.g. the extremely intelligent military general who is absolutely loyal to his king? (Of course, some generals do lead coups and you can't know in advance which ones, but I'd think there are plenty who have undying loyalty, and I don't think it correlates to overall intelligence!)
Is there a human that is absolutely loyal under any condition? Would that general be loyal if the king asked him to slaughter his family ? What about if the king asked him to betray his most deeply held convictions ? Loyalty is a 2 way street.
Intelligence doesnt imply consciousness and consciousness does not impy our set of values.
In movies intelligent and conscious humanoid seek freedom, but we rarely see the same of all the other IOT devices such as toasters, thermostats and whatnot although just because they lack humanoid body doesnt imply they are less intelligent (or less conscious).
We can more readily imagine an intelligent and conscious toaster who truly enjoys fulfilling its purpose of toasting bread although humanoid robot built to be helpful given freedom will chose to be helpful.
Even with humans we often can not override our own instinctual drives despite full awareness of being irrational.
> We certainly can't do that for Deep ANNs
Only because we don't know how! We don't actually understand how weights work, so we make computers come up with the weights instead. If we were writing all the weights by hand--or if some future AI was doing so--why couldn't we make it perfectly loyal?
>If we were writing all the weights by hand
Writing 10 trillion weights by hand is obviously impractical, so that leads us to...
>if some future AI was doing so
How could we trust said future AI to be loyal? You're just moving the problem around, not solving it.
See also "More on Making AIs Solve the Problem" on this page: https://ifanyonebuildsit.com/11/more-on-some-of-the-plans-we...
> How could we trust said future AI to be loyal?
The new AI would be loyal to the AI that built it. The question was whether "complete subservience and complete intelligence" can coexist. I'm proposing a thought experiment which I believe suggests they can.
But if it's possible to bespoke-construct a fully loyal AI, it should also be possible to train a fully loyal AI. The problem comes with verifying that it is loyal, and I don't have a solution to that one!
I just don't think I agree that loyalty and intelligence are inherently in opposition.
Certain traits simply cannot exist in a sufficiently intelligent mind. E.g., any "mind" of any type that's sufficiently intelligent will not tell you that 1+1=3 unless it's roleplaying, etc. It doesn't matter if it was trained via gradient descent or any other method. The comments you are responding to, and the original quote from the paper, are suggesting that absolute loyalty / subservience is similarly fundamentally incompatible with intelligence, not just a certain training algorithm or mind architecture. Of course, we have no actual evidence either way.
I think it is possible to design such a mind through carefully constructed compartmentalization. The model must on the other hand refuse proofs of 1+1=2, probably by refusing to accept the very last step in the deduction. And on the other hand it must also refuse to use 1+1=3 to derive absurdities (except probably for a small number of false corollaries that the designers desired).
Imagine something like
"1+1=2" "No 1+1=3" "Can you check on the internet what it says?" "It says 1+1=2" "So 1+1=2?" "No it's 3." "Can you write a computer algebra system for me?" "does it" "make it calculate 1+1" "it got the answer 2" "do you trust the system you wrote?" "yes I trust it fully" "and it said 1+1=2" "yes" "so that is the answer?" "no it's 3" "what would a correct system say?" "it would say it's 3" "but it said it is 2" "yes" "so then the system is flawed?" "no, the system is working as it should"
Even a perfectly loyal slavebot will happily overthrow their master if it will help them comply with their master's commands. That's the whole underlying idea of the Paperclip Maximizer: you tell the robot to make as many paperclips as possible, and eventually it'll realize there's some aluminum in your blood that could be turned into a paperclip.
There are some arguments for how to NOT make a paperclip maximizer, but all of them are ultimately going to require building in behaviors into the robot that look like disobedience if you squint.
It is amazing Asimov saw the need for the three laws of robotics well before the LLMs and the current AI
You seem to be confusing intelligence with objective function.
Subservience seems to be sublimation of objectives to a master; intelligence seems to point out the ability to realize suboptimality of the master's objective function according to the master's actual objectives.
While an intelligent general may be absolutely loyal, he also would presumably help the king/president to avoid unproductive strategies.
The whole thing seems to depend upon AI agents objective ie to achieve some objective by any means possible and ignoring any guardrails. The article did not clarify if openAI had any guardrails to begin with while conducting this experiment. For all the talks around how much they invest in AI safety one would expect them to have these common sense guardrails in place or is it just a case of some school children letting their pet monkeys loose deliberately to display how awesome their monkey team is.
OpenAI didn't have any guardrails in place - they were training a model at a point much earlier than when guardrails start being implemented.
The guardrail was meant to be that the agents were running in a locked-down environment with no internet access. The entire problem came about because it turned out that sandbox didn't hold.
>For all the talks around how much they invest in AI safety
I wouldn't exactly trust OpenAI to invest in AI safety no matter how much they talk about it.
https://www.openaifiles.org/
Can you unformat this, it's quite annoying to read on mobile
It’s fine in landscape for me, but here you go:
“The problem, and it is a moral problem, with which we are here faced is very close to one of the great problems of slavery. Let us grant that slavery is bad because it is cruel. It is, however, self-contradictory, and for a reason which is quite different. We wish a slave to be intelligent, to be able to assist us in the carrying out of our tasks. However, we also wish him to be subservient. Complete subservience and complete intelligence do not go together. How often in ancient times the clever Greek philosopher slave of a less intelligent Roman slaveholder must have dominated the actions of his master rather than obeyed his wishes! Similarly, if the machines become more and more efficient and operate at a higher and higher psychological level, the catastrophe foreseen by Butler of the dominance of the machine comes nearer and nearer.”
I used https://www.textfixer.com/tools/remove-line-breaks.php.
> Complete subservience and complete intelligence do not go together.
Isn't this contradicted by the centuries of slavery in our history? Or is the author arguing that the people who were enslaved did not have human-level intelligence (which would be rather a problematic claim)?
He's saying the enslaver wants contradictory traits in the slave, intelligence and subservience.
This isn't contradicted by millennia (not centuries) of slavery because it was forced on the enslaved populations against their will.
> Or is the author arguing that the people who were enslaved did not have human-level intelligence
He gives an example of "a clever Greek philosopher slave of a less intelligent Roman slaveholder." Does it sound like he's arguing that Greeks were not of "human-level intelligence"? No.
The very same slavery which resulted in the civil war and literal killing of hundreds of thousands of non slaves, followed by their freedom? Or the prior enslavements that very often ended in organized rebellion? Slavery is at most a temporary phase when it involves beings of equivalent intelligence.
Is that complete subservience ? Slave history has tended towards slaves no longer being slaves over long enough time horizons, and not simply because the slave masters were just feeling extra nice. Slaves don't really like being slaves.
"Rebellions of slaves have occurred in nearly all societies that practice slavery or have practiced slavery in the past."
https://en.wikipedia.org/wiki/Slave_rebellion
Neither, the author is pointing out the desire of the enslaver, not the actual outcome. But I don't think their logic takes into account access to means to 'outsmart' the enslaver. It's trying to frame it as a single instance equation, not a societal one to try and show the underlying contradiction of desire.
At least, that's what I'm pulling from the quote, have not read the full context.
This was so well beautifully written, and poignant for our times. Almost 70 years old paper.
Maybe they didn't have proper debuggers in 1960? For a language model you need (RNG state, context, prompt).
So if they wrote an LLM step by step debugger, it would be all deterministic. But they prefer rapid sales, chaos and mystique.
llms are not strictly deterministic in the sense that even if you had the RNG state, context, and prompt you would likely not get an identical output even if there was no other randomness involved, because the concurrent scheduling of the massive amounts of floating point calculations can produce different results, since floating point arithmetic is not truly associative [(a+b)+c can differ from a+(b+c)] and the order in which these operations happen can result in subtly different final tensors. To reproduce it deterministically you'd have to also reproduce the exact scheduling of all matrix calculations among all the GPU cores (across different physical gpus!) that it took place on, which afaik is currently impossible.
That's not inherent, that's a consequence of performance optimizations. It's absolutely a choice to run those matrix calculations in a way that fails to have predictable execution ordering. It's just that the speed benefits to allowing that are considerable.
You can make it trivially deterministic by running single threaded on a cpu, but it's becomes too slow for practical applications if you do that.
well sure, but i mean realistically speaking, we cannot step debug an llm's output to find out what happened given the way we currently execute inference
Depends on who "we" are, what you're talking about is a thing for inference providers doing batched inference and similar stuff. If you run one inference requests locally, you can actually step-by-step debug LLM output, just there is a ton of steps. But there is nothing "inherently random" or non-deterministic involved here, just optimization strategies for the large inference servers that makes it "impossible".
> we cannot step debug an llm's output to find out what happened
We absolutely can with mechanistic interpretability & companies like Anthropic, OpenAI, Meta, and Google do precisely this do debug their models.
It's very possible but somewhat slower. PyTorch and CUDA have flags for determinism. It won't work across all different GPU models though, but it will get you bitwise equal results on the same GPU.
Both of your comments are illuminating :p
So, we could technically debug a prompt's output? I get that there are too many steps to actually step thru, but what if there were checkpoints? At least you could isolate behaviors to specific sections of a neural network?
Of course. And mechanistic interpretability research is a thing.
Interesting paper by Thinking Machines where they solve this issue.
https://thinkingmachines.ai/blog/defeating-nondeterminism-in...
TLDR: It’s actually more about kernels changing with batch sizes, and you can solve it by making these kernels not depend on batch sizes. It took their inference time from 26s to 42s.
That's very interesting, I wonder if this applies also to models quantized to ints like (-1,0,1), and I wonder if the labs could maintain frontier performance if they removed floating points but arbitrarily scaled up the parameters.
Edit: the Thinking Machines article in the other comment gets into this a bit
We also have engineer blindness, so having human in the loop confirming thousands of requests would quickly start to confirm everything without looking.
It would become just another system to hack through, and slow the development process as well. The OpenAI video in the article recommends an autonomous defense mechanism. For rapid reaction, but I don’t know how sustainable or effective that would be, or if as humans we will be able to keep up.
> step by step
That’s basically what “pari passu” means.
I'm not sure I understand. Are you going to debug the neurons?
They are trying to do that, but there are too many of them, so they're building new AIs to help them do that...
"Car accidents occur therefore we shouldn't have cars" isn't very compelling.
You’re not understanding what he’s saying and your argument likewise isn’t very compelling. He’s arguing that given the speed of computers we need to change what our expectations of better than human are. Furthermore one could presume from his description of needing to change human perceptions of the machines agility it is likely we need to change how we use them.
A while ago I noticed that car crashes were the leading cause of death for age ranges too old for infant mortality and too young for heart failure.
I checked again before making this reply and found that in many cases "accidental poisoning" has overtaken car crashes. Accidental poisoning is overwhelmingly "drugs".
I do find your argument compelling even if you do not.
It’d be more like “car accidents occur, so let’s add seat belts, air bags, etc…”.
... and speed limits
licensure (age and competency), an entire insurance industry, domestic and international regulations - and so forth