This is the most boring comment posted on every one of these. NPM is by no means the worst offender here and offers a ton of solutions to this, lots of package managers are behind it or equivalent.
NPM gets targeted a lot because it's popular. That's it.
It’s correct that NPM is not unique but it is the worst for cultural reasons: no other ecosystem started with such a limited language, which lead to the culture of publishing tons of small packages working around things which everything else had builtin. A Python project which has a hundred dependencies is considered quite large but the median React project had north of 30 thousand for years and years.
It’s not that there’s a single stdlib feature which would’ve stopped this but more that JavaScript developers have been conditioned that it’s normal to install tons of packages and update them quite frequently so there are a lot of individual maintainers who if compromised have a surprising impact.
You’re exposed as a function of the number of dependencies so the communities which most normalize many rapidly updating packages are going to be at greater risk. That’s not a simple trade off — that enterprise Java app which updates on a decadal cadence is still worse — but it means you need to accept the risk and use other mitigations.
I just don't think that this is that unique to javascript, it's absolutely not about npm, and I don't think that this is well supported as a relevant feature that leads to these attacks.
Ok, I can agree it is a boring comment, but who is worse?
NPM gets targeted both because it is popular and because there is a wider attack surface (lots of little packages promoted by a huge variety of users) I have a high schooler who published work a couple weeks ago. This is good, but it comes with downsides. Maybe a couple more speed bumps or classifiers would be helpful. Maybe a consolidation of under maintained projects and deprecation is in order.
Arguably crates.io is worse. NPM has cooldowns and has for a while, it has had Trusted Publishing for longer, it has human-approved releases that separate CI/CD from actual publishing. Ruby is probably worse in every way.
Yeah, my point is just that other package managers aren't in a great spot. NPM even lets you separate out "publish" and "release" now where you can publish to the registry but you have to separately "ack" that to release. That's kinda a huge win if people use it.
I just think the framing that npm is so bad is really flatly invalid.
I don’t think npm itself is measurably bad in ways that are useful to index on. Like others, I think that JS as an ecosystem has a culture that encourages both dependency sprawl and living at the edge, both of which contribute to the end problem of OSS malware.
(Rust has a similar culture, to be clear. I don’t think it’s a death knell.)
Other than what happened with 'xz', which was upstream of it getting packaged, how many times has this happened in the debian packages system? Also very popular.
Debian's scale for package distribution is tiny and explicitly curated by maintainers. Everything funnels through Debian. The goals are completely different. Debian packages what's necessary for an OS, npm packages everything needed for any projects arbitrarily. Auditing scales to one of those, not the other.
The quantity of packages that exist at 162150 seems to say to me that there's a great many more than what are "necessary" for the operating system to exist.
sudo apt-cache stats
Total package names: 162150 (5189 k)
Total package structures: 145852 (6417 k)
Normal packages: 69505
Pure virtual packages: 1169
Single virtual packages: 64848
Mixed virtual packages: 355
Missing: 9975
Total distinct source versions: 71172 (1708 k)
Total distinct versions: 71172 (6334 k)
Total distinct descriptions: 139719 (3353 k)
Total dependencies: 435057/121405 (10.6 M)
Total ver/file relations: 73378 (1174 k)
Total Desc/File relations: 142692 (2283 k)
Total Provides mappings: 70533 (1693 k)
Total globbed strings: 268430 (6560 k)
Total slack space: 77.3 k
Total space accounted for: 47.0 M
Total buckets in PkgHashTable: 196613
Unused: 93545
Used: 103068
Utilization: 52.4218%
Average entries: 1.4151
Longest: 19
Shortest: 1
Total buckets in GrpHashTable: 196613
Unused: 86025
Used: 110588
Utilization: 56.2465%
Average entries: 1.46625
Longest: 7
Shortest: 1
162k vs millions. That's not even getting into package update velocity, authorship, the totally divergent goals, etc. I just think it's utterly pointless to compare.
No other package manager is worse than NPM. Outside of its 'popularity', there are several fundamental reasons why this continues to happen to NPM:
- Imported packages are not pinned by default.
- Typescript / Javascript's lack of a standard library encourages the developer to import more packages into their codebase to address the short-comings which increases the risk of importing a bad package.
- Post install scripts execute external code by default upon downloading dependencies.
All of this comes by default in the ecosystem and we continue to see more shai-hulud worms all easily targeting NPM. Not even signed packages are enforced by default either.
What do you mean? The lockfile of all package managers is there for pinning the exact versions. For yarn and pnpm, installs on CI run automatically from lockfile only, for npm I think you still need to run `npm ci` instead of `npm install`. But this guarantees that no new versions get pulled automatically in by CI.
> Typescript / Javascript's lack of a standard library
That's true, but it's not an issue of the package manager / registry
> Post install scripts execute external code by default upon downloading dependencies.
They are disabled by default in all package managers now, the user needs to manually allow them
This is the most boring comment posted on every one of these. NPM is by no means the worst offender here and offers a ton of solutions to this, lots of package managers are behind it or equivalent.
NPM gets targeted a lot because it's popular. That's it.
It’s correct that NPM is not unique but it is the worst for cultural reasons: no other ecosystem started with such a limited language, which lead to the culture of publishing tons of small packages working around things which everything else had builtin. A Python project which has a hundred dependencies is considered quite large but the median React project had north of 30 thousand for years and years.
I think that's barely meaningful. Which of the compromised packages would have been part of any reasonable stdlib?
It’s not that there’s a single stdlib feature which would’ve stopped this but more that JavaScript developers have been conditioned that it’s normal to install tons of packages and update them quite frequently so there are a lot of individual maintainers who if compromised have a surprising impact.
You’re exposed as a function of the number of dependencies so the communities which most normalize many rapidly updating packages are going to be at greater risk. That’s not a simple trade off — that enterprise Java app which updates on a decadal cadence is still worse — but it means you need to accept the risk and use other mitigations.
I just don't think that this is that unique to javascript, it's absolutely not about npm, and I don't think that this is well supported as a relevant feature that leads to these attacks.
> NPM is by no means the worst offender here
Ok, I can agree it is a boring comment, but who is worse?
NPM gets targeted both because it is popular and because there is a wider attack surface (lots of little packages promoted by a huge variety of users) I have a high schooler who published work a couple weeks ago. This is good, but it comes with downsides. Maybe a couple more speed bumps or classifiers would be helpful. Maybe a consolidation of under maintained projects and deprecation is in order.
Arguably crates.io is worse. NPM has cooldowns and has for a while, it has had Trusted Publishing for longer, it has human-approved releases that separate CI/CD from actual publishing. Ruby is probably worse in every way.
RubyGems actually adopted Trusted Publishing before both npm and crates.io. To my recollection, they were second after PyPI.
(I have no opinion about the overall security posture of these indices.)
No build script control though.
Yep. That remains the norm with Python source distributions as well. It’s a hard thing to overcome when it’s baked deeply into packaging assumptions.
Yeah, my point is just that other package managers aren't in a great spot. NPM even lets you separate out "publish" and "release" now where you can publish to the registry but you have to separately "ack" that to release. That's kinda a huge win if people use it.
I just think the framing that npm is so bad is really flatly invalid.
> the framing that npm is so bad is really flatly invalid.
Is it really though if we're getting thousands of compromised packages regularly?
You can do all the right things and still be legit problematic.
Yes, it has nothing to do with the design of npm (relative to similar languages/ repositories) and everything to do with the popularity.
I don’t think npm itself is measurably bad in ways that are useful to index on. Like others, I think that JS as an ecosystem has a culture that encourages both dependency sprawl and living at the edge, both of which contribute to the end problem of OSS malware.
(Rust has a similar culture, to be clear. I don’t think it’s a death knell.)
Other than what happened with 'xz', which was upstream of it getting packaged, how many times has this happened in the debian packages system? Also very popular.
I don't consider these comparable in any way that's worthwhile. The scale and goals are completely different.
How are they not comparable?
Debian's scale for package distribution is tiny and explicitly curated by maintainers. Everything funnels through Debian. The goals are completely different. Debian packages what's necessary for an OS, npm packages everything needed for any projects arbitrarily. Auditing scales to one of those, not the other.
The quantity of packages that exist at 162150 seems to say to me that there's a great many more than what are "necessary" for the operating system to exist.
sudo apt-cache stats
Total package names: 162150 (5189 k) Total package structures: 145852 (6417 k) Normal packages: 69505 Pure virtual packages: 1169 Single virtual packages: 64848 Mixed virtual packages: 355 Missing: 9975 Total distinct source versions: 71172 (1708 k) Total distinct versions: 71172 (6334 k) Total distinct descriptions: 139719 (3353 k) Total dependencies: 435057/121405 (10.6 M) Total ver/file relations: 73378 (1174 k) Total Desc/File relations: 142692 (2283 k) Total Provides mappings: 70533 (1693 k) Total globbed strings: 268430 (6560 k) Total slack space: 77.3 k Total space accounted for: 47.0 M Total buckets in PkgHashTable: 196613 Unused: 93545 Used: 103068 Utilization: 52.4218% Average entries: 1.4151 Longest: 19 Shortest: 1 Total buckets in GrpHashTable: 196613 Unused: 86025 Used: 110588 Utilization: 56.2465% Average entries: 1.46625 Longest: 7 Shortest: 1
162k vs millions. That's not even getting into package update velocity, authorship, the totally divergent goals, etc. I just think it's utterly pointless to compare.
left-pad was over a decade ago. it's a problem with the registry itself, more than just the package manager.
left-pad is totally irrelevant to this conversation.
No other package manager is worse than NPM. Outside of its 'popularity', there are several fundamental reasons why this continues to happen to NPM:
- Imported packages are not pinned by default.
- Typescript / Javascript's lack of a standard library encourages the developer to import more packages into their codebase to address the short-comings which increases the risk of importing a bad package.
- Post install scripts execute external code by default upon downloading dependencies.
All of this comes by default in the ecosystem and we continue to see more shai-hulud worms all easily targeting NPM. Not even signed packages are enforced by default either.
> Imported packages are not pinned by default
What do you mean? The lockfile of all package managers is there for pinning the exact versions. For yarn and pnpm, installs on CI run automatically from lockfile only, for npm I think you still need to run `npm ci` instead of `npm install`. But this guarantees that no new versions get pulled automatically in by CI.
> Typescript / Javascript's lack of a standard library
That's true, but it's not an issue of the package manager / registry
> Post install scripts execute external code by default upon downloading dependencies.
They are disabled by default in all package managers now, the user needs to manually allow them
Two out of three of your points - the first and last - are just incorrect.
Ruby is worse. crates.io is arguably worse.
echo "min-release-age=5" >> ~/.npmrc