Trolley problem. What's worse than all of humanity dead from ASI?
Sam Altman: "Development of superhuman machine intelligence (SMI) is probably the greatest threat to the continued existence of humanity. There are other threats that I think are more certain to happen (for example, an engineered virus with a long incubation period and a high mortality rate) but are unlikely to destroy every human in the universe in the way that SMI could."
most of humanity dead because some morons decided to try to kill everyone just in case the server farm evolved the inclination and the capacity to try to kill everyone...
Trolley problem: on the left track is a class of schoolchildren. On the right track is a a computer which a Harry Potter fanfic author argues has magic powers and the marketing department of Apple kind of agrees. Do you let Eliezer persuade you to kill the kids?