> In fact, the worst part about this dynamic is that these potential new backdoors will begin primarily useful for allowing the US to weaken its own systems, which will in turn allow foreign adversaries to find new ways to attack our communications. This deliberate self-sabotage will happen just at a moment when we’re finally learning how to defend our own infrastructure.
I don't understand how you can both argue for that law enforcement (and intelligence) agencies will force others to implement intentional backdoors AND also everyone will be using AI to find and secure ALL potential holes in the software so there won't be any vulnerabilities anymore.
Wouldn't one AI or another detect this deliberate backdoor and report it, as it'll look just like any other security vulnerability, the only difference being the intention?
I have respect for the author so I feel like I probably misunderstand something from the overall text rather than I somehow have a better perspective on this topic that the author knows very much more about than me. I felt like I nodded along all up until "So how is this a problem?" and now I'm not sure I understood correctly.
> Wouldn't one AI or another detect this deliberate backdoor and report it, as it'll look just like any other security vulnerability, the only difference being the intention?
That's precisely the author's point: deliberate backdoors will be more adversary-exploitable than ever before, but the demand for such from law enforcement agencies is likely to ratchet upwards.
> deliberate backdoors will be more
But that's one step after, I'm trying to understand how those backdoors even end up in software if everything gets automatically reviewed by the people working on these codebases? Wouldn't things like these be flagged by systems other than the developers tasked by the agency to implement it? How can law enforcement actually get these things implemented without big parts of the engineering team not seeing warnings about it happening?
I'm not sure you understand what is being described here?
If you're a US company building an app/device/etc. such that an intelligence agency like the CIA or FBI would want access to the data in that product which is normally secured, then they're not going to try to sneak it in there without your development team knowing. They're going to have a meeting with the owners of the company and say, "hey, we'd really like you to implement this backdoor for us, and in return we won't cause you in problems."
Note that this does certainly already happen a lot, but it's also not something that can happen across the board (like the author points out). Apple, being one of the largest companies in the world and who has one of their biggest selling points being their security, has explicitly refused to do this to the point that the intelligence agencies couldn't break into an iPhone until another company found a way to do so.
>I'm trying to understand how those backdoors even end up in software if everything gets automatically reviewed by the people working on these codebases?
The people working on these codebases are "in on it." Of course, we're talking backdoors which are very subtle, target very minimal infrastructure, and are known about by very few people. But, like I said, companies are currently shipping products with backdoors in them knowing they exist already. AI doesn't change that dynamic.
>Wouldn't things like these be flagged by systems other than the developers tasked by the agency to implement it?
The only people who would have access to the systems that can even be flagged by this stuff would be people who would know about it. Keep in mind that the scale we're talking here is massive. Think about how software development works at companies like Apple, Microsoft, Google, etc. There are devs working in offices all around the world where they only ever have access to a fraction of the code that company owns. These companies are very capable of keeping their stuff locked down. It's a necessary component of their work.
>How can law enforcement actually get these things implemented without big parts of the engineering team not seeing warnings about it happening?
Hopefully my explanation at this point is clear, but just to be concrete: backdoors are, by design, very hard to detect. That doesn't mean they're just sneakily written code that humans don't notice as they read over it, but, instead, they're very subtle implementations in very specific parts of huge systems that are already locked down to the point that the number of people who even have access to those portions of the systems are very limited. These agencies don't slip in backdoors without anybody noticing; they convince the minimal number of people needed to know about it to implement it. Again, we're talking about a meeting between the directory of a three letter agency and a CEO, where the CEO then directs the CTO to implement the backdoor who then instructs the handful of very high-ranking engineers to do so.
AI systems in these companies may very well flag these backdoors to the people who already know they exist, then these people can tell the AI "hey, those are their on purpose, so just move on," and the other 99.99% of the company will never know they exist.
I suppose it's important to emphasize, again, that these systems are incredibly massive and complex and most people at these orgs don't have any access to most of these systems, so it's not like you can expect an intern running BugBot across a repo and expecting it to find a backdoor.
I think people take the FBI or CIA too literally. I imagine they don't need to talk to owners and it's probably not even ideal. It might just be easier to get plants in the organization.
I would imagine most big companies, like Microsoft, have dozens of CIA and FBI plants in their organizations. Agents who are legitimate software engineers, tasked with acquiring intelligence and undermining security.
> If you're a US company building an app/device/etc. such that an intelligence agency like the CIA or FBI would want access to the data in that product which is normally secured, then they're not going to try to sneak it in there without your development team knowing. They're going to have a meeting with the owners of the company and say, "hey, we'd really like you to implement this backdoor for us, and in return we won't cause you in problems."
And then you say, loudly and publicly, "all the source code of our software is public, and our binaries use binary transparency so it's not possible for us to build a binary that doesn't match the source, and people will rapidly find this in our source code at which point we go out of business and you stop having a product to backdoor in the first place".
(And you move out of the US.)
And since this is a foreseeable future, you should start acting now to prepare for that future.
With laws like Chatcontrol and all, other jurisdictions are not necessarily any better.
In fact, we're increasingly seeing a desire to build the "backdoor" directly into the software, e.g. mandatory age verification, client-side scanning, etc.
> And then you say, loudly and publicly, "all the source code of our software is public, and our binaries use binary transparency so it's not possible for us to build a binary that doesn't match the source, and people will rapidly find this in our source code at which point we go out of business and you stop having a product to backdoor in the first place".
> (And you move out of the US.)
And then everybody claps.
Name me a software or hardware company—one big enough that the US government would actually care to force them to add a backdoor—that would be willing to give up the US market? The only one that's shown the least bit of backbone is Apple, and while I like them and appreciate what they've done in that vein so far, they're never going to move to open source software running their stuff, and they're pretty well embedded in the US, and very, very unlikely to try to move regardless of the headwinds there.
I'm fully with you that this would be a wise and moral thing to do, but frankly, our tech companies are neither wise nor moral. They are self-serving, greedy, and many of them have wanted to become the neofeudal overlords of a new order since before Trump started smashing the old one.
I suspect they want a backdoor that basically acts like a front door (current password regimes). That is, a kind of high level password that decrypts traffic given a specific, changing, password that only the government has access to.
It's seems like an odd-duck for sure, and I doubt it's a realistic proposition. I do think "perfect encryption for dummies" is all well and good until organized crime organizations are able to challenge the government in certain regions of our country.
All of this sends us deep into the realm of political philosophy, the nature and purpose of governments, and the freedom vs security tradeoffs we live with.
I listened to the latest Plain English podcast this morning, which was explicitly about the potential for a ransomwarepocalypse in the coming years, as open models let any tom, dick, or harry become capable of a plug-and-play ransomware attack, instead of that being left to the realm of professionals. It's a bit nerveracking to think about every nigerian prince scam suddenly becoming a sophisticated attack on your local water sanitation system.
https://youtu.be/vWvazbGPCCI
Ah, like new regulation requiring a "Portal For Law Enforcement" for platforms over X MAU or something?
I think I can kind of see that from the text re-reading it, it's kind of hidden though and not so explicit if this is what he is trying to communicate. Thanks for giving me a new perspective to read it with!
> In fact, the worst part about this dynamic is that these potential new backdoors will begin primarily useful for allowing the US to weaken its own systems, which will in turn allow foreign adversaries to find new ways to attack our communications.
The author agrees with you and addressed this point. The US forcing backdoors into its own systems ("own": Those for sale and distribution within the US) would create vulnerabilities making those systems weaker.
"backdoors" aren't holes, they are well-documented superuser APIs. The hack will be by compromising the API user's credentials (moles, stolen passwords, etc), not compromising the server's design intent.