We don't know exactly what the limits of AI improvement on our current infrastructure are, though. If the human brain is 20W, and a datacenter is 1GW, then maybe that datacenter can be 50 million times smarter than a human. If that's not already a risk to humankind I don't know what is.
Is the datacenter gonna grow legs?
Give me one datacenter, I'll keep it under control all by myself.
Now, if some dumbasses start hooking up their data centers to...I don't know, like--robot factories? That sounds like a risk to humankind.
All it takes for the AI to grow legs is to acquire money (should be no problem for an ASI) and pay humans to do what it wants in the real world.
And the ability to shrink itself to smaller-than-datacenter size.
If we could agree on that, no LLMs connected to robot factories, that would be a great place to start regulation. Though 1) I don't think we could agree on that 2) it's already well under way 3) we have extremist anti-regulation ideologues in control of american government.