> The problem isn't that AI could never be capable of causing real world damage. It is that there is absolutely no fucking path towards human extinction.
I think the "humanity will go extinct" discussion is a silly strawman being propagated either to try to spread fear, as the article suggests, or to make the anti-AI folks look silly, but it detracts from your first point which is that it may be "capable of causing real world damage."
We can be worried about real damage without having to defend the idea that every one of the planet's humans will die.
> "superhuman AI is unlikely to be much of a match against severing fiberoptic cable"
I think that the realistic scenarios all involve humans with an intent to do harm -- creating bio-weapons, finding vulnerabilities in infrastructure -- and using AI to help, so "severing the fiberoptic cable" doesn't apply.
Not to be too pessimistic, but it strikes me that in a worst case scenario, severing a cable might scale to severing global digital communications which is itself a massive worst case scenario even if we removed AI from the picture.
If the worst case outcome is we have to even temporarily shut down global shipping, banking, transport, health services, communications and national infrastructure to contain a self-propagating misaligned AI that can evolve itself and work its way into any sufficiently large computer system to escape humans preventing it from finishing its task, that seems like something we should be trying very hard to avoid.
Or you could argue that the super intelligence will turn us into a pet of some kind. Then the risk is we think we are controlling it while we are really the super intelligence's lap dog.
That is an argument that at least makes sense.
The super intelligence as homicidal maniac just doesn't make sense. There is less intelligent wildlife all around us and we mostly completely ignore it. We have more interesting things to do with our time as intelligent beings than carry out a bird or rabbit genocide.
It almost seems like the projection of some kind of paranoid delusion about change.