> And I really don't liek the idea of having the keys stored in the home directory in plain text.

so encrypt them.

or store them in a hardware token.

or on a USB stick (poor man's hardware token).

> There is also a risk of losing the keys if my laptop is damaged or gets stolen.

backups, full disk encryption.