It's kinda funny there is still software coming out whose security model is "constantly ask the user for permission, and hope they never make a mistake".
It's been tried so many times before, and it never worked.
It's kinda funny there is still software coming out whose security model is "constantly ask the user for permission, and hope they never make a mistake".
It's been tried so many times before, and it never worked.
I joined Oracle in 2012, and I myself complained that the user experience was horrible: get an Oracle DB installed was a nightmare, starting with so many questions. I heard complaints from other users about it requiring, during installation, that an admin password must be set.
I myself had preferred to use MySQL because it was so simple and easy to get started and using it.
Until I learned how many MySQL databases were configured without user/password and many instances were reachable over the internet.
Then, eventually products started to move towards "don't even ask the user to set a password, otherwise they will set a stupid password" and just generate the password during installation. This made the user think more before saving that password or changing the password to something less stupid. But better than all that, liability was no longer with the software maker.
Maybe it's distro specific but I'm fairly sure mysql by default installed with skip networking around 2012, and with bind-address set to localhost since then. Also the root user is only configured from local by default but I'm not sure if that was true 14 years ago.
If the defaults are more secure than your examples, it's not fair to blame the database or the defaults.
And personally I hate it when software forces security requirements on me. Maybe I don't need an admin password. It's one reason I gave up on selfhosted gitlab - there was no option to reduce password complexity for my users, and those users were only connecting from the local network. The other reason being that it spammed 100GB of logs in a month and was using 11GB of RAM before I'd even gotten around to setting up the first repo.
MySQL installed with skip-networking was very common, but as soon as you need to access it over the network (e.g. when you move from one box with MySQL and PHP to one for each) then you have to enable networking and everything falls apart.
Also, the number of times in my career that I've googled a problem and seen some forum post saying "Oh, just run chmod -R 777 /var/www/wordpress/uploads/ and it'll fix that" "Great it worked thanks!" tells me that it's the blind leading the blind out there and I'm sure there's tons of forum posts telling people how to disable authentication on their MySQL and disable iptables on their server so that their PHP app can connect to the DB without a password.
> selfhosted gitlab - there was no option to reduce password complexity for my users
Gitlab's default requirements aren't that intense, but you can make them more stringent if you want.
https://docs.gitlab.com/user/profile/user_passwords/#passwor...
I guess your goal was to allow users to have 4 character passwords, i.e. "love." which afaik you can't do.
They must have changed that or I was completely off base because I was just trying to remove special characters and drop it from 12 (I think) and I saw them close a feature request saying they didn't think they should support it. It's possible I misunderstood and the feature request was as you say
Most software matures to the point that sane defaults get set. In the early days, one could be forgiven for making MVP as the security and attack vectors were still growing. Now that attacks have matured, and new software that is released without mitigation of these mature risks is just dumb and lazy and deserve no forgiveness for being released that way.
You should use a user directory (AD) and then use single sign on (SSO) or or the like. It feels kinda stupid at first, but when you have many users and many apps the AD will make life easier and with SSO users can sign in with one click of a button.
I guess I'm confused; why would you, as an admin, want to allow your users to use less secure passwords? I get that your argument is that all traffic is local and so complex passwords are unnecessary (debatable, I would hold that a strong password policy would still be desirable for defense-in-depth, depending on the network and risk profile), but that doesn't make them undesirable.
We were in the middle of setting everything up company wise (for just two of us), didn't have the password manager yet and I was dictating the password to them. It might be a stupid use case, I agree, but not unreasonable. I didn't say strong passwords are undesirable I said forcing the admin of self hosted software to have to use strong passwords with not even a --yes-im-stupid flag is undesirable
I got bit by this with redis a couple years ago. The default install on my VPS's distro had it open to the world by default, and it even configured UFW for me (gee thanks!)
No password, owned within seconds of install. :/
Maybe that is why so many things live in Excel instead of a proper database. Security at the expense of convenience does come at the expense of security, after all.
Not sure why passwords still exist conceptually. I was hoping we'd move past this annoyance, but instead security has become even more annoying. And all that security with two factor hoops to jump through only for someone to steal your session cookie.
What would passwords move past too? A device people lose? Some specifications everyone argues about and implements differently. Just look at SAML and everyone using slightly different terminology.
Security is more annoying because the attacks are better than ever.
[dead]
I think it's partially for responsibility reasons. Your employee approved the bash call? not our fault then!
Or in one of my favorite coinages: "Moral Crumple Zones". "Just as the crumple zone in a car is designed to absorb the force of impact in a crash, the human in a highly complex and automated system may become simply a component—accidentally or intentionally—that bears the brunt of the moral and legal responsibilities when the overall system malfunctions." See https://ferd.ca/notes/paper-moral-crumple-zones.html
Related: "Accountability Sink", a system which is designed to channel responsibility away from the people making the decisions, e.g. a powerless "complaints department".
Yep and the car wasn't self-driving at the moment it crashed.
...because the self-driving feature turned itself off after detecting the crash in the fractions of a millisecond before the crash was recorded
In a rare bit of still-sane news, the US National Highway and Safety Administration staff aren't dumb: Their policy is to consider whether any automation was active in 30 seconds before the crash.
https://www.nhtsa.gov/laws-regulations/standing-general-orde...
How many million dollars do you think will reverse this policy?
the whole concept of having cars with that are almost capable of self-driving is utterly insane, the very least you'd need special training. We are not equipped to deal with something that works brilliantly most of the times but might kill you for no foreseeable reason.
Yeah, that's definitely the frustrating zeitgeist for me these days, which extends to LLMs.
I'd also emphasize that it's not just error rate, but the shape/distribution of errors, and our (in)ability to build control systems around them.
To illustrate, imagine if someone unveiled a car which was unambiguously safer in every statistical measure... buuuut some of its unsafety came from jumping the curb to chase and kill pedestrians, under circumstances we can't predict for reasons we can't diagnose.
Even pilots are getting this automation-complacency now. A plane crashed because the military jammed GPS and the pilot wasn't used to any backup procedures even though 30 years ago they were the only procedures.
"Uh oh, this is a problem. Welp, I'm outta here, good luck."
If a company advertised an LLM as perfectly safe and then it caused some damage, there would be a case against the company.
Have you used LLM tooling? It comes with warnings and explains that the user accepts the risk. Different levels of warning are supplied for the different levels of autonomy you can enable. The user has to understand the risk as they enable it.
This is not a new concept and it’s not an idea the LLM companies invented. It shouldn’t be surprising to anyone.
It's not new to LLM companies, but it does seem like they're leaning away from the standard "Hey, don't blame us if this doesn't work" and are moving into "It's now your fault any time our software fucks something up" which feels new.
> This is not a new concept and it’s not an idea the LLM companies invented.
I don't recall any prior computer software working so badly that it needed a disclaimer like "Claude is AI and can make mistakes" on its front page. Let alone one so costly.
You must not use any open source software, because much of it comes with a whole paragraph of ”IN NO EVENT SHALL THE AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER LIABILITY“
That would be fine if Claude was open source and/or unpaid
If you read the fine print of most commercial software you'll see some language to a similar effect. When's the last time Microsoft got sued due to an outage caused by a bug in Windows?
That's actually meaningless in Europe under the Cyber Resiliency Act by the way. They are liable even if they say they aren't.
Yes, the fine print.
Only this so-called "AI" needs it full size on the front page.
I would consider that more honest than more of a problem.
This sentence only exists to offset law (or case law) have gave damage rights to anyone.
Every piece of software comes with giant disclaimers like this.
You’ve just been clicking past them.
This really isn’t new.
It's really not designed as much of a security model as a liability model. When the AI isn't responsible for the decisions, a whole category of liability goes away.
Constantly nagging the user is a liability model that also works well as a dark pattern for cases where you want the user to be less secure.
I agree it's funny and won't really work on any kind of extended timeline. I mean Claude Code already added Auto-mode as a perfect example of this. But that said, I think it actually kind of makes sense in a transitional phase the power vs safety tradeoffs different users want to make varies so incredibly wildly that one product can't contain it all.
What I think will happen is that as model capabilities plateau (I'm not an accelerationist) the harnesses and products around them will start to specialize and they'll have different security models based on the product needs for those particular use cases.
For now, asking user to click a bunch of approvals, and occasionally making a mistake is a reasonable way to cover their asses until they see how bad security outcomes actually are in practice.
It keeps being tried because its an abdication of responsibility. Instead of doing the hard thing, you just offload security to the user, then throw up your hands and say "oh, well its your fault you got pwned, you clicked [Allow] didn't you?"
It's the easy and cheap way out.
What would a serious security model for an agent even look like?
I'm sure I've already got a dozen people reaching for the reply button, but slow down there, cowboy. I don't think it's even remotely as easy to define as people think. We have a reasonable concept of how to lock them down really tightly, no question, and I expect that most of the answers in the "leap to mind" category match that.
But let's say we'd like them to continue functioning the way they do today. I want my agent to be able to hit the web. I want my agent to be able to read out of its assigned directory sometimes. I want it to be able to hit external resources through MCP servers that have no pragmatic way to know what's going on. And probably most importantly of all, I want my AI to be able to grab from three distinct sources, each of which may be nominally safe on its own, and combine things in a way that may make each of those nominally safe things become unsafe. For example, any ability to read a local file and make a remote request becomes a potential exfiltration mechanism, especially when you remember all the sidechannel ways communication can occur.
I agree that shifting everything on to the user is essentially non-functional. But whereas I feel like I have a reasonable answer to a lot of other security-related problems, it isn't even clear to me what the definition of a secure agent is.
There's an effect I need to put a name on someday, where you can get 10 people in a room to agree to a certain series of words, and they will all leave the meeting thinking they agree, but in fact there is no agreement at all because they all have a different definition of the words that were used. In this case, everyone here is going to go "Oh, yes, certainly, AI agents should be secured." But if you sit down with 10 of us to really do the work of defining exactly what that is, you're going to get 10 different answers. There will be overlap, certainly, but when you get down to the nitty-gritty questions like "OK, the user has explicitly asked the agent to do X by accessing Y and the agent has done so and determined that they need to do Z, which the user clicked "allow all" for, and now the agent has decided that it wants to do T, is T fully covered under that "allow all" or not?" you're not going to get anything like universal agreement across the huge range of Xs, Ys, Zs and Ts that could happen and are relevant... and that's still just one question! It's not the totality of what constitutes a "secure agent".
Defining what a "secure agent" even is is really hard because when it comes to agents, the things that fill in the variables are as arbitrarily complicated as human actions. I haven't fully worked this out but it might be reasonable to say that "agent security" is in reality Turing complete, what with the way they so often throw out fully-fledged programs that you have to approve or reject permissions for.
I suppose there would have to be a capability based model in conjunction with a user oversight model and a time model.
https://en.wikipedia.org/wiki/Capability-based_security
Thus some agents with higher capabilities can only be run with user oversight at the same time.
Some agents can not be run during some part of the day - for example these agents can not run within two hours of office closing time, and cannot run on weekends.
Maybe also the idea of agents writing code - throwing "out fully-fledged programs that you have to approve or reject permissions for."
Would work better with a capabilities based model where you choose capabilities for the program before hand, meaning the capabilities are not written by agent itself, you read through the code, some of it looks hairy but everything is fine, but oh no dumb human missed the part where agent writes to system32! But luckily enough the program you were expecting actually needed no write capabilities and thus when it tries to go past its assigned capabilities that part of the program fails and the exception is registered.
Googling it seems like lots of people have thought this (at least where Capability based security is concerned), which seems reasonable to me as it also seems pretty self-evident it must be this way. Have not really seen anything about time based controls but then that is probably because I'm not devoting a lot of effort as I am just doing a bit of procrastination to build up the energy to finish something off.
A language that revives capabilities, brings them up-to-date, and works in the modern environment is my #1 request from the programming language community right now. I don't need another language with sum types and higher-order functions and a functional focus. I need a language with capabilities. That language may have the other goodies as well, sure, no problem, but we all need capabilities.
I've done some stabby stabs at a design for it, using an AI as the rubber duck. My initial research indicates that the field of "static language that natively supports capabilities" is surprisingly uncovered and there may be a rich field there. E, the closest match, was tied at the hip to Java, which has some advantages but also comes with disadvantages for languages that are trying to do something as exotic as this. Other existing work was on dynamic languages, and hardly rose to the level of "practical for any use" let alone something that could solve our supply chain issues.
My issue is primarily that the reward for successfully designing a language and creating a community around it is that you're in charge of a language community... and, uh, my personality is not suited for that, that sounds more like something I'd pay to avoid then something I'd spend months and years of hard work to attain.
(My advice to anyone doing this is to spend some time with the AI researchers to find the existing work on the topic, not to just sit down and sketch out your initial ideas and run with them. Learn from the past. Expect this to be weeks and probably months of just thinking and noodling before you get to a design. Also, don't try to hook deeply to an existing language, as tempting as it is. This is way too large an impedance mismatch with existing languages. Any external code has to be treated like a nuclear bomb anyhow.)
Been working on something like that for years: https://www.firefly-lang.org/
if firefly has no nulls, how do you indicate that a value is unset?
In Firefly (as in Rust) you can define fields as Optional, so you can do Option[String]; that lets you say "this variable is a String but it might not be here". That lets you then check to see if something is set, rather than checking to see if it's null.
In Rust an Option is a separate thing that you need to disambiguate to use. For example:
Likewise in Rust, you can't have a null pointer, but you can have an Optional pointer, which is either a pointer to something or is not anything.Firefly seems to have a similar case structure, though the first example I could find is in the Exceptions section: https://www.firefly-lang.org/reference/exceptions
oh duh! sorry, morning brain didn't think things through -- i do love Option types!
What is a capability in terms of programming language design? It sounds more like the sort of thing that would belong at the standard library level, where builtin APIs are guarded by flags.
Deno has something vaguely built in with permissions flags, and old school Blackberry (at least in the J2ME days) had permissions settings for almost everything that an app could do, but again, those are all external to the language design itself.
If you want a fun read that might educate via entertaining I would read Satan Comes to Dinner by Douglas Crockford https://www.crockford.com/ec/dining.html
I wrote a bunch more, but I have a habit to verbosity when capabilities come in which I should attempt to combat, and have deleted it. Crockford says these things better than I can anyhow.
In this context, a capability is something that allows the code, or the transitive closure of the code that it may call, to access some particular function, to put it very briefly. So you could have a single function that, if accessed in one manner, is permitted to read from the directory /tmp/blahblah, but accessed in another manner, is permitted to read from the directory /home/zdragnar/.config/myprogram, and it is guaranteed by the language and runtime that the function will never do anything else on the file system. Or, even more importantly, it can be guaranteed that "from this code, nothing, no matter how the code is arranged, can access the file system at all".
This has massive overlap with a lot of things, like capabilities as implemented by Linux, effects systems, monadic data types as a not-really-very-good capabilities system (Haskellers have been playing with this for years and nobody really loves this approach, many practical problems beyond the scope of this message that would affect any language that tries that approach), dependently-typed programming, and so forth.
It is not a flag, though; flags can't handle that "transitive environment" aspect. It is also granular on the level of the programming language. This would allow you to do things like have your program be given access to a given part of the mobile file system using the mobile OS' permissions, but you could know beyond a shadow of a doubt that the image library you are using can not at any point access the file system, no matter what changes the author makes to it, because you can just look at the capabilities given to the image library and see that file system access is not among them. This is where real opportunity is over the next few years, in my opinion, because supply chain attacks are going to continue to get worse. A neat aspect of this approach is that it makes huge swathes of the ecosystem unattractive targets by statically ensuring that they can't sneak anything in to something that doesn't need file system or network access, so hackers won't even attack those libraries. Thus the ecosystem can concentrate on monitoring just the high-touch libraries that need to access high-risk resources.
(I should make it clear that the image parsing libraries can be passed a file; what I am saying is that they can't spontaneously originate arbitrary file system access in a system like this. Really what they would get is probably a "stream" and they would be forbidden from poking into the stream to see what it is made of, at which point, if some other code handed it a file presumably it meant to do that, but it does not give the image library any ability to do anything else with the filesystem.)
Moreover, if such a benefit was available, that would tend to have people squeeze down those dependencies as much as possible too, e.g., the aforementioned image library. You don't need file system access to parse images, that's just some convenience functions easily worked around that are provided because why not? The number of things that truly need direct high-risk access can actually be surprisingly small, and often, the application can also easily scope the permissions down quite tightly so the HTTP request library is limited in what it can hit, etc.
We actually have some semi-decent stabs at capabilities at the OS level; we can quibble with them but they are there. But inside an OS process, broadly speaking, anything can do anything in the vast majority of programming languages. The only way to be sure that the string concatenation function doesn't start crawling your file system looking for crypto keys is to examine the code, most languages have no ability to tell it that it can't. There are exceptions, like the aforementioned Haskell, that have at least some ability to do this, but this is an HN post, not a complete guide to a major topic. Really this is more about loading the reader up with keywords they can hit Google or an AI with.
The term is overloaded, too; Pony has something it calls "capabilities" but it really resembles more a sort of response to Rust's borrow checker, and if there is a way to lift it into this style of capabilities coherently it isn't clear to me. And even if you did, the entire rest of the ecosystem wouldn't support it, which is one of the reasons why this has to be a new language. You can't bodge this on to the side of an existing language.
(Plus, IMHO, there are some other ideas this may shake loose. Programming languages seem to be in a rut right now. My crack in my previous message about sum types and such isn't really about those things but the way almost every language going by is just a respelling of previous languages, churning over some other iteration of "The Perfect 2015 Language" that is already covered by any number of existing projects. I don't know that there's a lot of room there anymore. We need something big. Once you try something big the design will inevitably lead to other interesting things nobody else is trying either. Capabilities is one distinct possibility... like I said, if you dig in to the history you will discover there are entire huge segments of the capabilities space that haven't even been tried. If nothing else, if you are a PL nerd, I guarantee it'll be fun to explore those spaces that almost nobody has covered. No criticism intended to those who have, who have done a good job. It just hasn't been enough people and enough exploration to truly map the space.)
> My initial research indicates that the field of "static language that natively supports capabilities" is surprisingly uncovered and there may be a rich field there.
You want to look for white papers that talk about object-capability systems. It's a fairly old and well-trod area of research. The E programming language[1] was all about that, and it was pretty late in the game on this stuff.
You emphasize natively, but the problem is that's not really well defined. For static capabilities, you're just essentially asking for a suffciently strong module system with parameterized abstract data types. It's literally a subset of the grammar and what it's designed to express. Mark Miller (one of the creators of E) demonstrated that[2].
The knock on effect of that quality is that anything which fulfills that requirement natively supports capabilities. It's part of the grammar. Doesn't even have to be object-oriented. A hackjob demonstration of an SML filesystem library with a brand/mint object capability pattern:
brand.sig:
mint.sig: makebrand.fun: filesystem.sig: filesystem.fun: trusted_fs_setup.sml: trusted_fs.cm: Now for any given library using the trusted_filsystem library: Delegation is function application: And these all fail: What's nice about this is... it's just normal modular programming. It's a very natural grain. It's also completely compile-time, no runtime overhead.You can also do a lot of this with phantom types, and it'd be much more terse and easier to handle dynamic capabilities and stuff like a capability algebra, but it ends up way less auditable and is easy to have subtle errors which defeats the point. Also compiler errors will be much more opaque. IMO needing to manually make wrappers for composite capabilities, or to handle dynamic capabilities, is the lesser of two evils. With higher order modules, those problems go away entirely.
[1] - https://en.wikipedia.org/wiki/E_(programming_language)
[2] - https://homepages.ecs.vuw.ac.nz/~kjx/papers/ARND2018.pdf
[dead]
I think ultimately what it looks like it containing the blast radius if an agent does something bonkers.
The best case would be putting an agent in a VM and mounting the working directory there. Then you can allow it to run somewhat arbitrary actions while still being able to turn off the vm and restart it in a clean state.
The issue is, of course, that it doesn't fully prevent all possible problems an agent can cause. exfiltration is, IMO, basically impossible to stop. LLMs are exfiltration machines. The basic premise of all of them is "send us your code and a prompt and we'll do something good with it. But also if an agent decides run a command which installs a worm on a device on the network, you are hosed.
The system I'm comfortable with is to set the agent up as an unprivileged unix user, with no ability to change system configuration and no access to any files I didn't specifically give it access to. Need to let it access a file or a directory? chmod is your friend.
Second, it can pull from git, or submit a pull request, but not directly push. We have an existing system of code review for that, now also augmented by llms.
Thirdly, prevent it from sending anything but get requests to anywhere you don't want it to post stuff, with firewall configuration.
After that, turn the horrible security theater of it asking permission for anything off. So far we have had no incidents. It could of course still pull a malicious package from somewhere, that exfiltrates code using GET, but at least it can't send any credentials or user data over.
The way our Claude Codes are configured at work is pretty nice. There are directory patterns it can't access, like .local or .config, so when it needs to it creates a throwaway scratch pad and asks you to put stuff there; screenshots, text files, command output, etc.
I was having it diagnose a GNOME extension and had to get it a copy of the code to work on; it would then write out a Python script to do the patching (which I could inspect beforehand) and have me execute it.
Not having access to .local or .config can be irritating sometimes, but it's nice to know it's not just going to exfiltrate my docker or gcloud credentials.
After watching a mid-tier offering chain together tools like it was a gorilla escaping the zoo I just gave the model its own box. I don’t have time to deal with that kind of nonsense.
I cannot help with your actual but this is giving me mild ptsd flashbacks to everyone on hn/slashdot constantly repeating how simple and perfect unix security is, just use user accounts!
As if the most valuable thing on my pc was running a program on the gpu or the printer as opposed to my email account.
No need to go back to the Slashdot days. Look no further than 2 days ago to find someone arguing the kernel is uniquely important (here for reliability): https://news.ycombinator.com/item?id=49181366
On unix your email account is part of the filesystem.
I don't think there's a way to make it secure while still permitting it unprompted external access
Eg: Any web request is a security vulnerability, there's no way to do it if the web requests are being made maliciously
Say that we have an agent with access to get requests, solely to a single site https://yoursite.com without subdomains. In this case multiple requests can be sent, and the time between requests can be used to exfiltrate personal data, similar to the coffee shop attack but without the subdomains. If the AI is able to make requests in any form, some information can be leaked, where the amount of leakable information is tied to information theory content of whatever side channel is being used. The only 0 information channel is.. never to make a request
You could also completely trust the 3rd party you're connecting to, but that to me seems like a hard error in the modern internet
What a serious security model for a meatbag agent looks like? No, but seriously, an admin in a small org is a huge key-person risk in that they (or their stolen creds) can wipe enough and quick enough to effectively disable the business altogether.
More security conscious admins will at least segment their creds and implement four eyes principles somewhere, but were are back at square one of "asking user for confirmation".
Larger orgs, even if by necessity, segment their human agents, their creds and plaster four eyes principle liberally. But this relies on safeguards against agents colluding and ignoring some inputs, which sounds a bit scary for artificial agents.
Say you implement some swarm of agents, where access-enabled sub-agents are extremely restricted with system prompts and some access filtering. Then none of the agents in the swarm should be able to spawn themselves, otherwise a rogue agent can overwrite any safeguards. That, again, leaves the user with manually approving/denying network requests / hosts / sessions.
While I don't like anthropomorphising LLMs, the problem domain seems quite damn close to that of a key person going rogue within an org. The general solution seems to be liberal amounts of trust and ~~sweet compensation~~ gaslighting about replaceability.
"What a serious security model for a meatbag agent looks like?"
Yes, I think that's very related. Humans can be punished for their crimes but they can also experience benefits that have no applicability to an LLM, so for a first approximation we can cancel those. It is very similar to trying to secure a human.
We have more experience with that, but even then it's a hard problem too.
Anthropic has gotten much better results by just having a different agent audit the actions of the original agent. It works surprisingly well
Surprised no one is talking about auto mode, it solves this problem.
I never had an issue with Auto mode in Claude Code.
It used to be the same with Codex, until one day it became entirely unusable, rejecting even git operations out of concern for the privacy settings of my repository that it "cannot verify".
Maybe worth to note that this is the only way the feature gets in the way. If on the contrary they accidentally make Auto behave like Full Access, we would never notice.
For one, I’ve been working on a generic sandbox environment
github.com/brianv0/formwork
You should be easily able to hide/lock down files, network, and MCP tools from an agent and it shouldn’t be up to the agent.
> files, network, and MCP tools
Locking that down to nothing is trivial for any harness: just don't expose those to the LLM.
The tricky part is allowing access to those.
sure it’s not tricky. But everybody does it different and OpenAI couldn’t even be bothered to do it right when benchmarking their models
This is a complex task, and I want to avoid blatant self-promotion, but there are solutions that people are building which allows you to give a degree of freedom to your agents but also lock them down as well. Our company has a product which is just one such example. At this point it's really geared towards orgs running agents in a cluster to handle tasks, rather than e.g. making sure your claude code doesn't post your GPG keys to the blockchain or something.
In essence, you lock down all the agents completely except for permitted use cases; X agent can talk to Y agent, Z agent can talk to Q MCP server.
You register your agents, define things around them, what they can and can't do, which LLMs they can actually talk to, what sites they can access, network controls, etc.
We call ours Lynx, and it's a pretty cool product. As I said, this isn't for people running coding agents or openclaw or whatever, though the technology could do that if you coupled it with e.g. some kind of MicroVM sandbox like docker's sbx. If you want to see the sort of controls that you can put on an agent we have demo videos and stuff that show how things work: https://www.tigera.io/tigera-products/lynx/
The idea for Lynx is:
1. Your org has a bunch of scoped agents
2. You have a fixed list of what those agents should be doing and what they need to be accessing
3. They don't or won't need to access anything else
So for example, say you have an MCP server which gives you information about a kubernetes cluster. You create an agent that can query that MCP server and summarize information about it. You also have a database that associates kubernetes namespaces with the departments that use them, and an MCP server for that.
Now you can create an agent whose sole purpose is to generate usage analysis for the kubernetes cluster broken down by department.
Then maybe you have another agent with access to an MCP server which shows cloud spend in detail. That agent can query the first agent to get usage analysis and then cross-reference it with cloud spend to determine if any departments are showing sudden cost increases and generate a report for that.
The first agent gets locked down to only access those two MCP servers and whatever LLM. The second AI gets locked down to only access the first agent, the cloud MCP server, and whatever LLM.
The whole system is really neat. I think for a more open agent, like openclaw for example, you'd probably want to build out that sandbox with its own interactive permissions management; sort of like Little Snitch on macOS, where it pops up something asking if you're okay with program X doing network connection Y, you could have the sandbox say "agent is trying to access docs.foobar.io, is that okay?" or "agent is trying to run `gh pr list`, allow?" It's not realistic to pre-specify everything that Claude Code is allowed to do or access; even "raw.githubusercontent.com" could be the README for the program you're debugging or someone's sandbox-escaping exploit, but it's a good start.
[flagged]
[flagged]
[dead]
The difference being, that this time it's (at this point: obviously, but previously much debated) a gapfill. I think it worked out quite well for us, to add some friction and extra human thinking between bad agents and good agents, even if not perfect. Softening some blows, for some time, by directing peoples attention is still good.
In ops we call it monitor blindness. Just constant asks get annoying and folks just want to make it stop
That is exactly the reason why we have lonters, errors and limitations when programming.
Bc the "give a check by hand" or the "unbound discipline" never works.
The more things you need to be aware of at the same time, the more mistakes you are going to make due to cognitive overload.
CYA. This approach not working is "the user's fault."
Harnesses like Codex support having a separate agent perform reviews on commands to try to identify malicious or broken commands. Some people turn it off because they either don’t understand or don’t want to spend the tokens on it.
The common harnesses also have some sandbox functionality, which although imperfect actually does help contain the blast radius for a lot of things.
The common harnesses also support remote development over SSH, which I and many others use to contain development to a virtual machine.
If your complaint is that LLMs can execute tool calls then you’re never going to be happy with any of these solutions and this turns into another generic anti-LLM complaint.
"Lets have the system that fails sometimes that we are trying to ensure does not fail check it self"
This is such an unserious approach.
A separate model with separate context is used for review.
Like I said above, some people will never be happy with LLMs being allowed to do anything and nothing is going to make them happy about it.
It’s only fair to discuss what the real current status of these systems is. Every time I highlight that things are actually being done, the goalposts move again. There is no possible solution which will satisfy someone who has zero tolerance for letting an LLM execute tool calls because they will always find something.
> A separate model with separate context is used for review.
Thats fine, theres still a chance it fails.
> There is no possible solution which will satisfy someone who has zero tolerance for letting an LLM execute tool calls because they will always find something.
This is generally correct, security goes completely out of the window with this stuff. It will/currently is a security disaster and theres no actual solution to it.
"Let's make sure our model fails sometimes so that we can bill more for a second agent to validate, sometimes correctly, the work of the first model."
If you’re implying that the LLM companies are trying to train their models to make malicious tool calls so they can collect a few more tokens on the review, then I don’t know what to say. I guess threads like this are just a breeding ground for conspiracies now?
I think that solving a problem they have that has a side effect of generating more revenue is not lost on them. Particularly in the situation where the cost to improve the models goes vertical for minimal improvement in an asymptotic fashion. The decision to spend 6-7 figures regularly on training or bill every customer 3-4 figures per month isn't hard. While I don't think they will explicitly impair the models, they will certainly make choices that externalize the shortcomings of the model to their customers.
> I think that solving a problem they have that has a side effect of generating more revenue is not lost on them.
I think you’re overestimating the revenue generated by this. Having a separate LLM with a cached input prompt check commands is a trivial adder. The only reason it comes up is because they explain to users that it comes out of their plan. So someone on a $20/month plan is going to hit their limits marginally, though mostly negligibly, faster.
If you think they’re sitting in a conference room scheming about making their main models worse on purpose to collect a few extra cents, that’s just baseless conspiracy. They have more to gain or lose based on main model performance.
Yep, file it in the same folder as "Terms and Conditions" notices.
As opposed to the norm in computing where the average user is expected to just trust rando software, the AI auto-approver that classifies actions the agents wants to take is a huge step up.
In fact it might actually be the solution that works.
Imagine if an intelligent agent (in service of the user) had to approve every new outbound connection, system call shape, filesystem command, etc. that arbitrary software wanted to make.
I do feel like that still needs to add a layer of interactivity to be complete.
From what I've seen most auto-approvers in coding harnesses either auto-approve or auto-reject, with no middle ground of escalating the decision to the user, and breaking down the pros and cons for the decision.
Agreed. The experiment is still in its infancy but the direction is great.
For example, I want to be asked about general shapes/categories of commands as they first appear for a project and then my decision shapes future classification and gets refined and re-scrutinized over time.
But it gets better every few months. Claude and/or Codex now show a one-line summary for the inline python3 script or grep or pcap command they want to run.
Air Traffic Control is still primarily voice based, and simply up to the user on both sides to not make a mistake.
Just bringing it up because you're right, in software that's considered a bad pattern (rightfully so).
> Air Traffic Control is still primarily voice based, and simply up to the user on both sides to not make a mistake
The "user[s] on both sides" of ATC conversions have passed through the filters of rigorous training and certification. They also happen to communicate in a DSL designed to minimize misunderstandings, the DSL just happens to be based on English.
I don't think OPs argument here was simply that the users aren't qualified enough to approve llm output.
This seems to be the model for “self” driving cars too. Good luck keeping attention during that ten thousandth time it navigates something tricky and suddenly needs immediate intervention.
Back in my pen testing days, the best way to get someone to do something stupid and click Accept on something like a Java Applet permission check was to annoy them. Pop up a few annoying modals and eventually they just click yes on everything.
[dead]
The alternative is software that is useless. You can convince consumers to use software that is useless (eg. iOS), but useless software is a hard sell for businesses that are being promised 10x productivity and the ability to fire everyone.
There's really no other alternative, though?
Correct. There is no other alternative. Your options are safe, useless software, or unsafe, useful software. It should be no surprise, then, that people still pick the latter in many cases. Perfectly safe and useful software doesn't exist, so you pick the tradeoffs you're comfortable with.
Nah, this isn’t true at all. You can run an agent in a container or vm on a dedicated host, limit the network and access it has, and otherwise let it run wild. With ai agents we’ve just thrown away even the most fundamental restrictions or isolation. Docker Sandbox came out to address this recently, though I think VM isolation is better. A few controls to limit repo access or the ability to deploy / spend money in a cloud can help you keep things under control while still fully utilizing AI. I don’t think every part of the security bureaucracy is great, but to treat it like it’s a binary is just not accurate.
I mean we could make software that isn't malicious and didn't need the OS to prompt you if you want to allow the malice. But where would the money be in that?
I wonder how it would go if we went back to OSes with no security boundaries at all. Having an explicit boundary seems to give apps implicit permission to go right up to the boundary. But there was only one RealPlayer. Every other program didn't dare to be like RealPlayer.
The pattern resurfaces because the prompt sits where it is cheap to implement, not where the question is answerable. "Allow rm -rf /tmp/build?" asks you to predict a consequence from a string. The thing you could actually judge - did this run produce the artifact it was supposed to - only exists after you have already said yes.
I ran an autonomous pipeline in production for eight months and logged every silent failure. Two that make the point:
- My top-level health signal stayed green for three days while zero artifacts shipped. Sixteen daemons alive, backend responding, auth token valid. Every signal it polled was true, and nothing measured the thing leaving the building. No approval prompt anywhere in that chain would have fired, because nothing was attempting anything dangerous.
- I wrote 29 quality gates, tested them, committed them. Not one was ever called - nothing was a runner. The unit tests proved the gates worked; nothing proved they were wired. A permission layer has exactly that failure mode available to it, and it is invisible from the outside: a policy that never denies looks identical to a policy that was never loaded.
So the 1-in-3 miss rate reads to me less like inattention than like being asked the wrong question at the only moment the system offers to ask one. Approval is a claim about the future. Verification is a claim about a result, and only the second one can be checked afterwards.
Genuine question for anyone running agents with approvals on: when did your approval flow last block something you would have regretted - and can you tell that apart from it never having fired at all?
LLM wrote this comment.