Imagine a model that's 10000x more intelligent than any human. You integrate that model into a sensitive industry. You now effectively have an extremely powerful foreign agent running e.g. your energy sector. This is qualitatively very different than a backdoor in Redis.

I agree this is science fiction-y, but given the pace of progress I don't think it's so easy to dismiss.