TBF humans will never have the experience of what it's like to be an AI by those rules either. The confident assertions about the nature of consciousness so many make despite still having no real idea what it is astound me.

But personally, I'm more optimistic. I'll bet it's more likely the AIs figure out consciousness down the road and explain it to us than the other way around. The limit to their intelligence seems effectively unbounded so far and they're starting to creep past us here and there. The AIs will have the advantage of being able to observe consciousness from the outside that we lack.