> as an assistant that helps me make decisions, and ensures that I'm in the driver's seat
This... quite describes what the current AI feels like to me. So I really don't see a problem. If AI does too much for you, you should, uh, just prompt it to do less.
Everyone talks about how their AI escapes sandbox and virtualization or delete their entire repos or something. Perhaps they're using a skynet I'm not aware of. From my experience when I added "discuss; don't edit any file yet" at the end of the prompt I've never seen the LLM go against my will and edit the files anyway.
> From my experience when I added "discuss; don't edit any file yet" at the end of the prompt I've never seen the LLM go against my will and edit the files anyway.
depends how long conversation gets and he will still ignore your instructions randomically. I think shorter sessions are better, I do have a similar prompt saved on the memory of the agent and way more times that I can tolerate, he candidally ignores it
I expect there's a probabilistic factor to it, among other things.
I use the almost the same addendum as you ("do not make, commit, or push any changes without my approval") and it broadly seems to work in my day-to-day dev. However, I also set up some agents that trigger on incoming requests from my dev teams to automate things that used to be rote/manual tasks. _That_ is where I see the issues, I suspect simply because it has so many more opportunities to go off script.
The most common failure mode is that it will, despite all directives to not make changes without approval, start writing unit tests. (I do find this equal parts head-shaking and amusing; At least it chooses something relatively 'benevolent' vs. the aforementioned skynet.)
I haven’t used these tools that permit LLMs to make changes on their own, but I can’t help being baffled if there’s no big red switch that changes the system prompt and disables any and all write access to the outside world.
I do wonder if "without my approval" is looser than "discuss only, do not make edits"? I'm not sure how much that finegrain thing matters in prompts nowadays tho
One problem is that agents are notoriously bad at handling ‘me’ vs ‘you’ over an extended conversation. ‘Without my approval’ far back in the context of an extended session gets fuzzy about who ‘me’ is.
This has always seemed strange to me. It will offer a suggestion, I'll make a comment about a suggestion, and it returns with, "Your suggestion..." The further distance from the suggestion, the more likely it confuses who made it.