I don't whitelist IPs for ssh anymore, but I always run sshd on randomly selected port, in order to not get noticed by port scanners.
I do it for a really long time already, and until now I am not sure if it has any benefit or it's just umbrella in a sideways storm.
As long as you understand it's security by obscurity, rather than by cryptography.
I don't think it's wrong, it's just not the same as eg using a yubikey.
This won't hide you completely, but it will reduce log spam.
My sshd only listens on the VPN interface