At the risk of sounding extremely dumb, I have a question for you: if the hardware is susceptible to something that you can't actually reproduce with the software everyone runs on it, who should care, and why? Is it even really fair to call it a vulnerability at that point? Is the idea that this is supposed to help identify a different mechanism of exploiting the vulnerabilities with the shipped OS too?
To give an analogy, it almost feels like removing the protection circuitry from a Li-Ion battery and then testing if it can catch fire, and observing that it does. Should it really worry users?
> that you can't actually reproduce with the software
If the vendors of that software are not aware of the failure mode then they may accidentally introduce it by later code changes. Systems evolve. Blind evolution leads to pernicious and widespread failures.
Great analogy. Li-ion batteries have several layers of defense against exploding, one of which are vents that, if all else fails, let the hydrogen gas safely escape rather than building up. It's perfectly fair for independent testers to say "we haven't found any flaws in the protection circuitry yet, but we should bypass it to see if the vents work as designed".
I'm not disputing that it's fair to investigate that. What I'm asking is if it's fair to then call it a vulnerability without establishing that the thing is, in fact, vulnerable as a result.
I would say it's like calling the battery a fire hazard if the vents don't work, but actually that's not analogous because the necessity for vents doesn't merely arise from the need to protect against bad design of the protection circuitry. They're needed for safety even if your circuitry design is flawless. So the analogy is actually kind of poor in that regard.
This is why a distinction is often drawn between vulnerabilities and exploits — many more things can be weaknesses in a system that can only be exploited in combination with other vulnerabilities.
An obvious example is web browsers, where a vulnerability can easily be uninteresting because it lives in a sandboxed process… until you find a sandbox escape, then it is critical.
As long as you suspect there may be other vulnerabilities in the other layers, it is worthwhile investigating and fixing them, because defence in depth only works until someone manages to put together a full chain.
Not the author, but as someone who frequently has to answer this question, I'll chip in:
A mistake is a mistake, whether you have a way to reproduce it right now or not. It's pretty much a given that whatever means you have right now to reproduce the problem will evolve and broaden the scope. Also, if you haven't found a way to reproduce the problem, it doesn't mean it doesn't exist: it takes a lot more effort to prove that it's impossible to reproduce than to simply not being able to reproduce the problem.
As long as we're using analogies, let's use a car one. If your car gets into a car crash, you want it to be safe and not explode, yeah? Would you rather drive around, and see if it happens to explode if you happen to get into a crash, or would you rather setup a test arena, crash as many copies of your car, as many times as possible, to see if it ever also explodes. And then, once you've shown it can explode when the left back window is halfway down and the passenger door is halfway open, then from there it's easier to figure out how those set of circumstances might actually happen in the real world.
It's research, which often involves a ton of work for zero pay off. It's usually thankless and unrewarding, on the off chance that there is some exploit to be had.
The real benchmark is whether it can run Doom while measuring why Doom runs.