I would be terrified of the server being breached and having my passwords/key material leaked.

Well the passwords are GPG encrypted, so they'd need your private key too, not just breach the server. What sites you have accounts on is visible though.