The issue with the direction of technology is it tends to make things that were once very hard to impossible. This is why there are bumper stickers mocking Libertarians by saying things like "legalize recreational plutonium".

If we are not careful and suddenly release tools with far more capabilities than we expect you go from "smart" people being able to do it, to some angsty teenager being able to pull it off in their bedroom.

The current issue with AI is we are squirting out new models faster than we can complete long term testing on them. We'll find new model capabilities long after they've been in the field. Just assuming safety is how you catch cancer from your food dye, or how you change the atmosphere around you and start burning down your planet. Now just imagine that involving intelligence and doing with a few percent of the worlds GDP to make it happen.