In every of these threads there's a bunch of snarky comments, either acting like this class of attack is exclusive to npm, or that nothing has been done about it. I don't think that's fair.
There's plenty of comments mentioning delay lines, and the other good stuff pnpm (and others) have implemented in response to protect package consumers.
That bit that's getting less conversation is the tools on the package maintainer side:
- MFA for publishing: https://docs.npmjs.com/requiring-2fa-for-package-publishing-...
- trusted publishers, available for about a year: https://docs.npmjs.com/trusted-publishers
And as of recently, staged publishing, essentially combining the best of both those features: https://docs.npmjs.com/staged-publishing
Now you can: - Publish from CI, without static credentials
- AND require a maintainer to approve it using MFA before it actually goes live to the registry
If you want you can still use something like GitHub Actions Environments protection to require multiple approvals, or a time delay, on the CI side.
We need to encourage the community to adopt these publishing protections or this will continue to be an issue.
> - MFA for publishing: https://docs.npmjs.com/requiring-2fa-for-package-publishing-...
> - trusted publishers, available for about a year: https://docs.npmjs.com/trusted-publishers
According to [1] "All affected packages were published via GitHub Actions OIDC from the RedHatInsights/javascript-clients repository, indicating the upstream CI/CD pipeline itself was compromised."
So the malicious package would have gotten the happy little green star, with users assured it was "Built and signed with provenance."
[1] https://lwn.net/Articles/1075742/
they need to make it required for everyone, and then they'll have done something
My read is that there's a crowd that is unimpressed with mechanistic changes when in their view there is a cultural issue.
From the outside looking in, web dev has this frantic wild west energy to it. Mutability, dynamic typing, standards changing constantly, frameworks changing constantly, continuous delivery, CDNs, live A/B campaigns, large numbers of dependencies, sensitive user data spread out across a lot of infrastructure.
I'm not saying that's an accurate view and I don't think "I told you so" is the right attitude, but I can understand the place it comes from.
> In every of these threads there's a bunch of snarky comments, either acting like this class of attack is exclusive to npm, or that nothing has been done about it. I don't think that's fair.
I mean it keeps happening lmao. You can track npm attacks these on a calendar. Someone made a npm parody of the classic "no way to avoid this" The Onion article.
It's great there's work to stop it all but also... it keeps happening. I find it funny in a "here we go again" way.
IMHO those are both lipstick on a pig solutions. Ultimately all this stuff is just a variation of "make releases harder to publish", which isn't going to do anything but train people to evade them. Notably, neither would have prevented the xz-utils backdoor from reaching package distribution, which remains the gold standard for sophisticated upstream compromise.
The bug here isn't that we need to better authenticate already-trusted upstreams for packages, it's that the upstreams cannot be trusted as the sole source for security at all. Upstreams are a bunch of hackers[1] who aren't really interested in, nor will ever be good at, solid release engineering practices.
But some people are! The solution in the Linux world (and the one that saved us from xz-utils) is that there is a second level of human beings responsible for reviewing, auditing, packaging, and customizing those hacker-generated upstreams for the benefit of their users. These people have different eyes, different consumer requirements and different quality metrics. And they catch bugs and malfesance that the upstreams aren't prepared to do.
NPM (and cargo/PyPI et. al.) continues to think it can short circuit this requirement for human labor. It can't.
[1] In NPM's particular ecosystem, a bunch of web jockeys used to extremely fast release processes, loose compatibility requirements, and extreme reliance on reuse. This really explains why we see this with node packages more than Python or Rust: older and more conservative programmers just don't have as many rakes to step on.
> The solution in the Linux world ... is that there is a second level of human beings...
AKA "unpaid labor". I don't think that's a good solution, either. Certainly it's only by pure luck that no malefactors have infiltrated the ad hoc, anonymous social proof communities that Linux depends on, and I don't think other systems should emulate it.
The real solution (for Linux too) is a paid package curation service. Or really, a small handful of them competing on price, speed, reliability.
> ... a second level of human beings responsible for reviewing, auditing, packaging, and customizing those hacker-generated upstreams for the benefit of their users.
> The real solution (for Linux too) is a paid package curation service. Or really, a small handful of them competing on price, speed, reliability.
That was also what I was thinking aloud a moment ago. And there would be a business opportunity, too. Perhaps not like RHEL et al. full-blown stuff per se, but say smaller scale guarantees with different pricing; web, AI, scientific computing, and whatnot. At the pace things are progressing, I'd guess you might even get desktop etc. users on board (for nominal pricing).
There is a version of that. It is called RedHat Enterprise Linux. : )
> Certainly it's only by pure luck that no malefactors have infiltrated the [pinko commie Linux hippy commune]
Yeah... no. Sorry, that's a wild misunderstanding of the economics of the Linux ecosystem, modern libertarian thought and the employment status of people with write access to the packaging layers.
But there is a second level of people reviewing packages on npm. They're the ones that report issues like the github issue this HN thread is linked to, and they very frequently get malicious npm packages taken down within a day of publishing. The big issue is just that not everyone is using a cooldown to avoid packages less than a day old and so people who install new packages at unlucky times don't get the benefit of that layer of review.
I don't understand why you're confident that those Github issues won't just end up coming later if literally everyone adds this cooldown
The security companies looking for and reporting the issues aren't going to use the cooldown too.
> they very frequently get malicious npm packages taken down within a day of publishing
If I'm reading the secondarily-linked blog post correctly, this was live for 12 days before discovery.
> Notably, neither would have prevented the xz-utils backdoor from reaching package distribution, which remains the gold standard for sophisticated upstream compromise.
Mandating that the final binary is compiled without having any access to any test file though would have prevented the xz-utils backdoor as it was conceived though.
A proper packaging setup would first verify that all the tests are passing and happen in an isolated environment. And that isolated environment either returns which tests failed or gives the green light.
When the greenlight is given (that all tests are passing), another environment should first delete all files related to tests and then build (in a bit-for-bit reproducible way btw and we're basically here already so that's good) the final binary / package.
If you prepare your final package in an environment that has access to test files, there are simply way too many ways obfuscated binary data can be hidden in test cases / test files.
I'm not saying the NSA (sorry, Jia Tan) wouldn't have tried something else but I think we should really move to build/packaging that discards non essential data/files before compiling.
P.S: note that as a side-effect of reproducible builds... If we have reproducible builds and if we add, later on, a builder/packager that discards tests files and ends up with a final package that's not bit-for-bit identical to the package created while having access to the test files during the build, we've just detected a backdoor hidden inside test files (like the XZ utils one). As a really mindboggling food for thoughts: if we were to recompile all the Debian binary packages that are already reproducible today (95% of them), but while discarding all tests files before the build, we may catch other backdoors.
> Mandating that the final binary is compiled without having any access to any test file
It would, but I'm not seeing that as a suggestion? That was a very clever side channel for hiding the build-time payload. It wasn't remotely the "root cause" of the exploit, which was that a malicious actor got write access to the release process of trusted software. I mean, if you can do that, you can surely find other clever ways to hide your junk.
To wit: you're not wrong, you're just stuck on minutiae. By all means make the case, but at best you're proposing a small constant factor optimization.
Why doesn't Java seem to have anything close to this issue? Isn't it a solved problem?
No deep dependency graph, so easy to audit and inspect. Stable development process, so easier to manage the flow of updates. And I believe a more cautious ecosystem. Not everyone is rushing to create or adopt new libraries, especially for what could have been a single file. So most libraries are about solving a domain, not just one single algorithm.
Same thing with C, Perl, PHP,…