It's actually worse than this because it assumes alignment as a concept even makes sense. For example:
If the the Chinese government asks their ASI to create a bioweapon against the West, should it? No, presumably not – an aligned AI would be one which disobeys the Chinese government even if they created it.
Okay, so what if the US government asks their ASI to help it in one of their wars instead? Would an aligned AI kill humans on the order of the US government? No, again, presumably not.
So what have we have we even created here? An AI which is more intelligent and powerful than us which also doesn't take orders from us?
Is this what most people thing of as alignment and is this what humanity actually wants?
We should stop using the word alignment. It's a BS term for a concept which simply cannot make sense if alignment is both to mean an AI which we control and an AI which will not harm us.
I think it is worth noting that the article addresses this.
It touches on it, but it doesn't address it.
They talk about "value alignment" but fail to define what those values are – is it aligned to the values of the US government, or are they suggesting they want to build an AI with it's own values so it can decide for itself when and how it will intervene in wars and other human affairs? And again, is an AI which disempowers humanity in this way aligned? Many would say no, although as I argue, disempowering humanity is probably better than the alternative if we can assume it's roughly aligned with our interests (which we obviously can't because it's super intelligent, but that's another issue).
Fundamentally the problem here is that humans don't have an aligned set of values you can align an AI to.
The moment you start defining what alignment actually is in practise you simply must accept it will be unaligned with the values of others. There is no getting around this and hand waving around the issue isn't good enough.
They should tell us explicitly what they're trying to build.
No, what I mean is this part of your comment is unrelated:
The article already introduces terms to distinguish between those two concepts ("goal alignment" vs "value alignment").You are right, of course, that when we are discussing value alignment, it becomes very relevant whose values the AI is supposed to be aligned with.