I tried to raise that with my internal security team recently - don't clutter my vulnerability dashboard with issues in dev dependencies. They somewhat rightly pointed out that malware needs to be dealt even if it's a dev dependency. So my suggestion went nowhere because I guess we can't filter by type of vulnerability.
Working in the EU energy sector where we have to work with NIS2 compliance, I'd argue that your security team rightly pointed it out. I suspect that's what you mean though, and the rightly is just there because you agree with it but don't like it. We work with even more tight dependencies policies than just having alerts. We have a set of pre-approved and yearly vetted packages, like pandas or pyarrow for Python data work. Aside from that we have some isolated development environments where your pipeline can get access to something like SQLC for Go. Which is essentially where your dev dependency lives in it's own environment where it can produce the code it needs to and then submit it for approval into your regular dev environment.
Ironically we'd probably need to run Dependabot itself in a mirrored environment since it too has external dependencies we'd probably not want to vet.
I do think external dependencies are among our biggest security threats though. It's so hard to vet them, and compliance basically comes down to "We trust the apache software foundation enough, and pyarrow is vital to our business, so we accept the risks", and then you lock versions and aren't the first to update except for vulnerabilities. Shadow AI is obviously the number one security threat right now, especially in enterprise with people who are very tech savvy. This makes dependencies so much worse though, because now everyone can (if their systems aren't locked down tight) do so many crazy things. Both with the "non-sanctioned" AI but also with the code it can generate for them.
> compliance basically comes down to "We trust the [third party] enough, and [third party component] is vital to our business, so we accept the risks"
This is the beginning and end of reasonable security. This is what it's always about, and if you go beyond it, you risk practicing art for the sake of art, at the expense of customers and other stakeholders.
Security is about understanding and managing risk. Not about achieving some mathematical perfection (which is actually only achievable by making your system an inert piece of rock, but people realize that way too late, after piling on way too many pointless "security improvements").
I've almost always gotten everything I want from security teams over my career. Usually a quick and honest chat with them gets you pretty far.
I always lead with some flavor of "In my perfect world, I have 100% access and ability to everything, everywhere all the time. In your perfect world, I don't even have a computer. Here's why I need X permission / Y user group / z application"
From the perspective of big corporate security - developers are a wild nuisance who file the lions share of the tickets and soak up an inordinate amount of resources. Being able to at least explain to them that you understand their objectives and are not overrequesting just for the sake of overrequesting goes a long way.
I'm not the previous poster, but in my experience you can get a lot of mileage out of having dev teams (tediously, at least the first time) go through all potential vulnerabilities, decide how risky each one is based on likelihood and impact, and then get them to address the high/highs somehow (e.g. by upgrading a dependency, or writing extra code to guard against the issue, or fixing the issue if it's a home-grown vulnerability).
First, this is great reply with lots of real world experience to share.
> I do think external dependencies are among our biggest security threats though.
This sounds like a good business opportunity. I know that Sonatype has a business to vet Java dependencies. Does your company use it? I am guessing that Sonatype may be expanding into other open source ecosystems.
Companies like Sonatype would be an issue since they are owned by USA private equity. We would not give "Vista Equity" access to anything with the current EU US relationship. It's bad enough that we're so tied into Microsoft, which the EU might task us with leaving if they deem it critical enough for the security of the European energy sector. That's a risk we live with though, there isn't a realistic alternative.
That being said, our current strategy is more along the lines of building thind within standard libraries. We really wanted to adopt Go company wide, but it's proven impossible for non-SWE staff to use AI to create their projects in anything but Python. So instead we've created AI configurations that know our security policies, the tools we want them to use and we've setup security policies which won't even allow you to run a Python executionable inside a virtual environment unless your devices is sepcifically allowed to do so in that specific folder. Similarily we've completely limited what VSCode extensions they can use down to the named folder version. Which sort of sucks, and I doubt a lot of it would be possible if it wasn't because the c-levels are personally liable for security under EU law.
We'll see what happens after september when the summer holidays are over and the real token cost of AI will kick in.
> First, this is great reply with lots of real world experience to share.
I know how they came about with this setup, but I think that's the wrong way of approaching the problem.
Their problem is legacy and trickle-in features in an otherwise unmaintainable code.
With AI, they can rewrite their software to minimize dependencies and in general reduce the attack surface by allowing the business to automate more on their own.
But it requires bold management decisions and people in position of authority that can pick the right battles for the advancement of their careers.
Of course, all the generated code has to reviewed and vetted for by a senior developer. Of course, this has to be re-done every now and then when new classes of vulnerabilities appear that the previous generation didn’t have in mind.
Or do you just trust the AI that was trained on a lot of bogus code?
Yeah I completely understand their intent, but I might get 30 vulnerabilities across a multiple repos flagged in a week. It is already tedious to check them all and assess if they're worth worrying about let alone having to update them. These are 99% Javascript though - I suspect other ecosystems are much more manageable.
It's easier to keep stuff up to date these days. If you have a project with typescript, unit tests, and end to end tests like cypress you can just have dependabot create the PRs to update packages. If everything passes you just have to hit the merge button.
Just updating everything is probably easier than assessing if it's possible to trigger an exploit with the way you use the package.
This is exactly how developers of malware want you to behave. Update without really thinking about it.
I do wonder how long it will take before an attack is developed by submitting a semi-genuine vulnerability, shortly followed by a ‘fix’ including malicious code.
Notably this does nothing to "solve" the attack vector. You've got a live bomb in front of you and you're adding 10s to the countdown hoping that _others_* find it and defuse it in that time period.
I would challenge anyone proposing this to define more than one party doing security checks on packages to prove the point that many projects are waving their hands nebulously around saying "security vendors" and then YOLO'ing code into their codebase because they didn't here the muses wailing.
Alternatively from the other direction - Point to any dependency in your project. How can you get *POSITIVE SIGNAL* that security vendors _did_ look at it and okay it? How much scrutiny did they put into it? At what version did they last inspect it?
With today's AI glut of tokens, multiple someones are scanning security checks against the changed code. The real problem, as was before, was getting anybody anywhere to pay enough attention for long enough.
Only if their scanning detects it though. Malware authors have incentive to figure out how to fool the tool. They don't even need to be right all the time, any attack that survives works for them, and creating accounts is easy.
Yep this is what has happened to small teams. You really only have time to approve the dependabot changes and go go go. Otherwise you'll never get anything productive done.
The other option is to simply ignore updates and do them on a schedule, e.g. once every 1-2 months.
Or you take the alternative approach of flattening and minimizing your dependency graph. Having so many dependencies you can't reasonably field bug reports in them is a chosen tradeoff, even if it doesn't feel that way.
Dependencies and supply chain attacks are probably the greatest risk to a lot of software orgs, as they run them across all their environments: Development (with secrets and other valuable artefacts on developer VMs), CI/CD pipelines which may have access tokens to production (and other) environments, and production itself.
Notably even security companies are being impacted by this[0]. The scale of these attacks has amplified quite significantly the past three years, but are not solely exclusive to the javascript ecosystem [1] or even just namesquatting/typosquatting [2].
The resolution is broader security awareness, "onion layered" security controls and implementing simple non-burden inducing processes and policies. Sometimes not updating (what was wrong with the previous version of a dependency if there was no immediate vulnerability or production issue caused by it?) or having a two week cool down for updates (which some supply chain tooling natively supports) can appease some security functions through clear communication of the supply chain risk etc.
If anyone has interest in courses aligned to your org on improving developer and broader engineering management awareness on this, e-mails in my profile :).
I unironically think the solution is vibecoding your own Javascript blobs that use no frameworks and have no (or minimal) external dependencies. At this point it is entirely feasible for many kinds of projects.
Everyone talking about malware in dev dependencies as if dependabot only raises issues about that, but it does not. It raises warnings about all sort of "vulnerabilities" irrespective of the threat model.
Even worse, it incentivizes randomly updating dependencies, which is what actually allows supply chain attacks.
Dev dependencies is how they compromised SolarWinds and thereby most of the US federal government.
> The attackers used a supply chain attack. The attackers accessed the build system belonging to the software company SolarWinds, possibly via SolarWinds's Microsoft Office 365 account, which had also been compromised at some point. SolarWinds was using build management and continuous integration server TeamCity provided by the Czech company JetBrains. In 2021 The New York Times stated that unknown parties apparently embedded malware in JetBrains' software and through this way compromised also SolarWinds.
I don’t know what kind of software you write, how valuable your company’s infrastructure is, etc. But supply chain and insider threat in security/infrastructure is a big topic — that I’m sure they’re concerned about because that’s their area of responsibility.
Even if I’m personally sympathetic to not wanting to deal with the churn of dev dependency updates.
This is very real, but such CVEs are such a tiny fraction in relation to denial-of-service-due-to-regex that it’s hard to take the system seriously.
So far as I’m concerned the solution is to isolate everything as much as possible. I’d love to see something on the CVE classification side to also address the signal to noise problem but I don’t see it happening.
So I have a library and its ultimate purpose is converting globs to regexes. Someone sent me a ReDoS vulnerability report with a 4.0 CVSS score because if you write an obscene glob pattern you'll get a correspondingly obscene (and inefficient) regex. What else would you have it do!?
Pretty much - I don't know too much about the CVE process but if ReDoS stuff was flagged at the CVE level as "exploitable only with unconstrained inputs" then great - I know my tests have sane inputs, so I'll close thanks.
Vulnerable dependencies are very different to compromised or backdoored dependencies though. Noone's taking over Solarwinds because their build tools had a ReDOS involving input from their own config files.
I got reminded every week that my static site generator "Jekyll" is insecure.
Ok. Hacking me by changing the input to my Jekyll rather involves being on the other side of the airtight hatch.
I tried to raise that with my internal security team recently - don't clutter my vulnerability dashboard with issues in dev dependencies. They somewhat rightly pointed out that malware needs to be dealt even if it's a dev dependency. So my suggestion went nowhere because I guess we can't filter by type of vulnerability.
Working in the EU energy sector where we have to work with NIS2 compliance, I'd argue that your security team rightly pointed it out. I suspect that's what you mean though, and the rightly is just there because you agree with it but don't like it. We work with even more tight dependencies policies than just having alerts. We have a set of pre-approved and yearly vetted packages, like pandas or pyarrow for Python data work. Aside from that we have some isolated development environments where your pipeline can get access to something like SQLC for Go. Which is essentially where your dev dependency lives in it's own environment where it can produce the code it needs to and then submit it for approval into your regular dev environment.
Ironically we'd probably need to run Dependabot itself in a mirrored environment since it too has external dependencies we'd probably not want to vet.
I do think external dependencies are among our biggest security threats though. It's so hard to vet them, and compliance basically comes down to "We trust the apache software foundation enough, and pyarrow is vital to our business, so we accept the risks", and then you lock versions and aren't the first to update except for vulnerabilities. Shadow AI is obviously the number one security threat right now, especially in enterprise with people who are very tech savvy. This makes dependencies so much worse though, because now everyone can (if their systems aren't locked down tight) do so many crazy things. Both with the "non-sanctioned" AI but also with the code it can generate for them.
> compliance basically comes down to "We trust the [third party] enough, and [third party component] is vital to our business, so we accept the risks"
This is the beginning and end of reasonable security. This is what it's always about, and if you go beyond it, you risk practicing art for the sake of art, at the expense of customers and other stakeholders.
Security is about understanding and managing risk. Not about achieving some mathematical perfection (which is actually only achievable by making your system an inert piece of rock, but people realize that way too late, after piling on way too many pointless "security improvements").
I've almost always gotten everything I want from security teams over my career. Usually a quick and honest chat with them gets you pretty far. I always lead with some flavor of "In my perfect world, I have 100% access and ability to everything, everywhere all the time. In your perfect world, I don't even have a computer. Here's why I need X permission / Y user group / z application"
From the perspective of big corporate security - developers are a wild nuisance who file the lions share of the tickets and soak up an inordinate amount of resources. Being able to at least explain to them that you understand their objectives and are not overrequesting just for the sake of overrequesting goes a long way.
Asking out of curiousity - how would you or how does your org handles this right now?
I'm not the previous poster, but in my experience you can get a lot of mileage out of having dev teams (tediously, at least the first time) go through all potential vulnerabilities, decide how risky each one is based on likelihood and impact, and then get them to address the high/highs somehow (e.g. by upgrading a dependency, or writing extra code to guard against the issue, or fixing the issue if it's a home-grown vulnerability).
First, this is great reply with lots of real world experience to share.
This sounds like a good business opportunity. I know that Sonatype has a business to vet Java dependencies. Does your company use it? I am guessing that Sonatype may be expanding into other open source ecosystems.Companies like Sonatype would be an issue since they are owned by USA private equity. We would not give "Vista Equity" access to anything with the current EU US relationship. It's bad enough that we're so tied into Microsoft, which the EU might task us with leaving if they deem it critical enough for the security of the European energy sector. That's a risk we live with though, there isn't a realistic alternative.
That being said, our current strategy is more along the lines of building thind within standard libraries. We really wanted to adopt Go company wide, but it's proven impossible for non-SWE staff to use AI to create their projects in anything but Python. So instead we've created AI configurations that know our security policies, the tools we want them to use and we've setup security policies which won't even allow you to run a Python executionable inside a virtual environment unless your devices is sepcifically allowed to do so in that specific folder. Similarily we've completely limited what VSCode extensions they can use down to the named folder version. Which sort of sucks, and I doubt a lot of it would be possible if it wasn't because the c-levels are personally liable for security under EU law.
We'll see what happens after september when the summer holidays are over and the real token cost of AI will kick in.
> First, this is great reply with lots of real world experience to share.
I know how they came about with this setup, but I think that's the wrong way of approaching the problem.
Their problem is legacy and trickle-in features in an otherwise unmaintainable code.
With AI, they can rewrite their software to minimize dependencies and in general reduce the attack surface by allowing the business to automate more on their own.
But it requires bold management decisions and people in position of authority that can pick the right battles for the advancement of their careers.
Of course, all the generated code has to reviewed and vetted for by a senior developer. Of course, this has to be re-done every now and then when new classes of vulnerabilities appear that the previous generation didn’t have in mind.
Or do you just trust the AI that was trained on a lot of bogus code?
Yeah I completely understand their intent, but I might get 30 vulnerabilities across a multiple repos flagged in a week. It is already tedious to check them all and assess if they're worth worrying about let alone having to update them. These are 99% Javascript though - I suspect other ecosystems are much more manageable.
It's easier to keep stuff up to date these days. If you have a project with typescript, unit tests, and end to end tests like cypress you can just have dependabot create the PRs to update packages. If everything passes you just have to hit the merge button.
Just updating everything is probably easier than assessing if it's possible to trigger an exploit with the way you use the package.
This is exactly how developers of malware want you to behave. Update without really thinking about it.
I do wonder how long it will take before an attack is developed by submitting a semi-genuine vulnerability, shortly followed by a ‘fix’ including malicious code.
The cooldown setting in dependabot solves this attack vector. By setting it you give security vendors time to scan new packages.
Notably this does nothing to "solve" the attack vector. You've got a live bomb in front of you and you're adding 10s to the countdown hoping that _others_* find it and defuse it in that time period.
I would challenge anyone proposing this to define more than one party doing security checks on packages to prove the point that many projects are waving their hands nebulously around saying "security vendors" and then YOLO'ing code into their codebase because they didn't here the muses wailing.
Alternatively from the other direction - Point to any dependency in your project. How can you get *POSITIVE SIGNAL* that security vendors _did_ look at it and okay it? How much scrutiny did they put into it? At what version did they last inspect it?
With today's AI glut of tokens, multiple someones are scanning security checks against the changed code. The real problem, as was before, was getting anybody anywhere to pay enough attention for long enough.
Only if their scanning detects it though. Malware authors have incentive to figure out how to fool the tool. They don't even need to be right all the time, any attack that survives works for them, and creating accounts is easy.
Dependency cooldowns fix most of those problems.
Yep this is what has happened to small teams. You really only have time to approve the dependabot changes and go go go. Otherwise you'll never get anything productive done.
The other option is to simply ignore updates and do them on a schedule, e.g. once every 1-2 months.
Or you take the alternative approach of flattening and minimizing your dependency graph. Having so many dependencies you can't reasonably field bug reports in them is a chosen tradeoff, even if it doesn't feel that way.
In agreement with frodd above.
Dependencies and supply chain attacks are probably the greatest risk to a lot of software orgs, as they run them across all their environments: Development (with secrets and other valuable artefacts on developer VMs), CI/CD pipelines which may have access tokens to production (and other) environments, and production itself.
Notably even security companies are being impacted by this[0]. The scale of these attacks has amplified quite significantly the past three years, but are not solely exclusive to the javascript ecosystem [1] or even just namesquatting/typosquatting [2].
The resolution is broader security awareness, "onion layered" security controls and implementing simple non-burden inducing processes and policies. Sometimes not updating (what was wrong with the previous version of a dependency if there was no immediate vulnerability or production issue caused by it?) or having a two week cool down for updates (which some supply chain tooling natively supports) can appease some security functions through clear communication of the supply chain risk etc.
If anyone has interest in courses aligned to your org on improving developer and broader engineering management awareness on this, e-mails in my profile :).
[0] - https://socket.dev/blog/ongoing-supply-chain-attack-targets-...
[1] - https://orca.security/resources/blog/hades-pypi-supply-chain...
[2] - https://checkmarx.com/zero-post/python-pypi-supply-chain-att...
I unironically think the solution is vibecoding your own Javascript blobs that use no frameworks and have no (or minimal) external dependencies. At this point it is entirely feasible for many kinds of projects.
Everyone talking about malware in dev dependencies as if dependabot only raises issues about that, but it does not. It raises warnings about all sort of "vulnerabilities" irrespective of the threat model.
Even worse, it incentivizes randomly updating dependencies, which is what actually allows supply chain attacks.
A lot of the recent npm attacks have been exfiltration from dev machines, which would just as likely from dev dependencies.
Developer's machines and cicd systems are high value targets. They were absolutely right to point that out.
Only in the castle and moat security model popularized by Microsoft and the various "security" vendors that leech off it.
And the money wasted on the security theatre around this outdated concept is astonishing.
Dev dependencies is how they compromised SolarWinds and thereby most of the US federal government.
> The attackers used a supply chain attack. The attackers accessed the build system belonging to the software company SolarWinds, possibly via SolarWinds's Microsoft Office 365 account, which had also been compromised at some point. SolarWinds was using build management and continuous integration server TeamCity provided by the Czech company JetBrains. In 2021 The New York Times stated that unknown parties apparently embedded malware in JetBrains' software and through this way compromised also SolarWinds.
https://en.wikipedia.org/wiki/2020_United_States_federal_gov...
I don’t know what kind of software you write, how valuable your company’s infrastructure is, etc. But supply chain and insider threat in security/infrastructure is a big topic — that I’m sure they’re concerned about because that’s their area of responsibility.
Even if I’m personally sympathetic to not wanting to deal with the churn of dev dependency updates.
This is very real, but such CVEs are such a tiny fraction in relation to denial-of-service-due-to-regex that it’s hard to take the system seriously.
So far as I’m concerned the solution is to isolate everything as much as possible. I’d love to see something on the CVE classification side to also address the signal to noise problem but I don’t see it happening.
So I have a library and its ultimate purpose is converting globs to regexes. Someone sent me a ReDoS vulnerability report with a 4.0 CVSS score because if you write an obscene glob pattern you'll get a correspondingly obscene (and inefficient) regex. What else would you have it do!?
Pretty much - I don't know too much about the CVE process but if ReDoS stuff was flagged at the CVE level as "exploitable only with unconstrained inputs" then great - I know my tests have sane inputs, so I'll close thanks.
These DoS Regex 10/10 CVEs in some minor helper function in some package that is used once in some random side code pathway are so damn annoying.
If I could filter out DoS CVEs‚ I would.
Vulnerable dependencies are very different to compromised or backdoored dependencies though. Noone's taking over Solarwinds because their build tools had a ReDOS involving input from their own config files.
It's even worse if you're SOC2 because then you actually have to go through and mark them as "not exploitable." The noise is insane right now.