> Eliezer and Soares and their ilk are possessed with the fear that we’ll have a superintelligence that’s A) utterly narrowminded and pigheaded in pursuing psychopathic or megalomaniacal goals, while at the same time being B) incredibly deep and broad in understanding the universe and how to get things done – yes, this could theoretically happen, but there is no rational reason to assume it’s likely!

At first, it does seem rather unlikely to get both A) and B) from the same mind. But if we look at humans, we certainly find highly intelligent humans (on the higher of the scale of B), who have low emotional intelligence and empathy, which is, when you think about, it at the core of A) (someone with very low EQ and empathy, who is also very wealthy, and a true believer in their own intelligence boosted by a positive feedback loop from those around them since such people tend to surround themselves with sycophants, is going to exhibit megalomaniacal qualities. We need look no further than Elon Musk for a current example.

If we can find this easily in humans once they get to the point that they have achieved great power, it's not difficult to imagine finding it in an AGI system created by humans and in the image of a "human with superpowers".