> but on actual identity that needs to have the access.

Not quite. You shift the trust from the key bearer (the most interested party in all of this) to the identity provider.