The point of the paper isn't that embeddings contain information, it's that even if you don't know what model generated a set of embeddings you can still recover information from the geometry of the point cloud itself.

The fact that this is possible also adds some pretty strong restriction to the set of possible maps you could use to remove that information. No linear map will work since all embedding spaces are ~an orthonormal matrix apart, so some form of encryption is necessary. This wasn't known until very recently.