Don't know whether this is a common outcome, but I tried the "remove the walls" example, and the result was... scary. It completely changed the game so that movement is now diagonal, and made the arbitrary decision that up/down move you on the positive diagonal, and left/right move you on the negative diagonal.
The problem, of course, is that having only the one single "you can't win" law is severely underspecified, but the solution was too clever by half, and highlights the problem with this approach — every program will be under-specified, because, at some point, writing the laws becomes a bigger problem than writing the code itself.
This becomes a real issue because the combination of underspecified but rigid laws pushes the aI towards this sort of "creative" solution that matches the letter but not spirit of the law. In this case, the issue was obvious, but I seriously worry about what sort of shenanigans will occur in less obvious cases.
Exactly, "you can't win" is grossly under-specified. The goal of the demo is just to show that laws can't be broken. Yet, if that's your only law, the AI can do whatever to protect it - including changing how the character moves, or even removing the flag entirely!
So, yes, that's the issue with laws: they only protect what you remember to write. They're not a silver bullet. But they can still be incredibly useful, because it takes a small law to protect against entire classes of bug, covering your entire codebase. For example:
LAW: "the sum of all balances in this contract must be zero"
This one liner would have prevented Ethereum's infamous "The DAO" hack, where millions were stolen and almost undermined the entire project. But if your laws are under-specifying or ill-specifying your intents, Bend can't do anything to protect you.
> LAW: "the sum of all balances in this contract must be zero"
What prevents AI from writing an arbitrary calculation that makes sure that sum is always zero while breaking a whole other bunch of assumptions?
If the answer is more laws, then this does not address the parent comment.
Thought about automated discovery of laws in an existing codebase?
If you can find a law which the existing code obeys, and show it to a human, and if they agree, save it. And maybe the AI could make a decent guess as to what kind of laws would appeal to a human versus which wouldn't – a simple law identifying a fundamental constraint the system obeys is good, something really complicated or constraining something coincidentally true isn't
Or some kind of measure of coverage? you'd never want 100% – then your laws would become so complex you couldn't follow or maintain them – but if it is very low, that can be a signal to increase it
Discover the laws in a current code base?
They are usually in the “tests” folder.
So anything capable of extracting unit tests is extracting “laws”.
One might consider the test name the text of a law. And the AI fills in the details.
From the codebases I have seen the "laws" are scattered between unit tests and lots of little tests and assertions in the actual code, plus cultural norms and "do it like this" patterns.
But a good start could be found in the unit tests.
I don’t think that’s right.
Unit tests check whether a specific code module is implemented correctly given its (implicit) specification.
Good “laws” are independent of the structure of the code.
Tests may be a useful source for inferring what the laws are, but much of the actual content of the tests aren’t “laws” at all.
you might be interested in property-based testing, which somewhat enumerates tests based on invariance and induction
I think the premise is more that if one is given a grouping/constellation of unit tests that one can derive generalized laws by looking at what behavior those tests were checking for or against.
if there are a dozen unit tests trying to determine if some regex can escape a sensitive area, then one can derive a generalized 'don't let the regex escape from here' type rule -- or at least you could theoretically. I'm sure in reality that'd be a big minefield much like harness self-skill-writing has been.
Yeah all of these attempts to try and control AI outputs by using language fail to grasp that language is itself the problem. It is a closed system with no ground truth. Words only point to other words in an endless self-referential chain. There is no "closure" in language, ambiguity cannot be erased.
Imo LLM researchers would benefit a lot by reading what continental philosophers have said on the topic of language. Barthes, Sarte, Deluze, Derrida, et. Al have a lot to say that explains why we're running into the same problems over and over.
Natural language is ambiguous. Specifying exactly what you want in a way that is truly verifiable is the act of coding.
Start printing the t-shirts!
You mean t-shirts of this comic, from 2016?
[0] https://www.commitstrip.com/en/2016/08/25/a-very-comprehensi...
Bend would make Dijkstra happy even when proof checking can’t verify if the laws are what was actually meant.
I actually think Asimov is more instructive here, while Gödel and Tarski tell us the tool can’t prove itself…
Nonetheless, it is a worthwhile endeavor and I hope more rigorous practices like this catch on.
I think LLM researchers understand how LLMs work and what the limits of using natural language as an interface are. The problem is everyone else thinks they're basically magic, expecting them to be infinitely intuitive but also strictly deterministic, like the computer from Star Trek.
The computer from Star Trek sounds like it could entirely plausibly have been made with LLM components and a century of harness tweaking piled on.
[dead]
> writing the laws becomes a bigger problem than writing the code itself.
But that's how it is anyway, no?
Defining clear boundaries and clear goals is the hardest part. If you get those right everything else is rather trivial.
Well if possible you want an AI that understand the spirit of what you are asking for and will add all the missing stuff, instead of an AI that just want to hack its way to the result. Kinda what Fable brought to the table. For instance as a simple example I ask it to change the text that shows the email of the user by his name and Fable did all the code in case there is the family name missing etc. That this last part you want an AI to do. Helping you to build the system with you and not gaming what you ask for for reward.
Side thought - I like the idea of this as a game, where you’re essentially fighting a monkeys paw / tricky genie. Not totally sure it’d work but I like the concept of trying not to get caught out.
I got the same result when I tried the "remove the walls" example. I followed up by telling it to reimplement up/down/left/right movement without reinserting the walls and it basically made the square with the flag "unenterable". Like with a force field.
Respectfully I'm not sure if I share your worry though. You would have the same problem if you wrote extremely thorough / exhaustive unit tests or extremely precise types and didn't allow the LLM to amend them. You're basically shrinking the state space of what is considered a "correct" program per your spec. That the LLM has to get extremely creative to break your program is a _feature_; in my experience, an LLM does not have to get creative at all to break a typical program written in an enterprise setting, and that's unfortunate.
> writing the laws becomes a bigger problem than writing the code itself.
Of course because at its limit programming is basically defining desired behaviour under all circumstances and logical conditions.
Heh, We're writing Asimov in real life after been warned to decades.
Well described the problem scope.
I wonder if harness-hooks + Jev (equivalents) could semantically lint for `sloppy_law` etc when ever they are edited