You could restrict the ssh port by ip as well.