Access to the server gives you access to the encryption keys, unless the server is just storing end-to-end encrypted material for someone else and doesn't do anything with the data.