The preferred solution would be something like a Yubikey. However:
> just go to any old computer and ssh into my server
You've typed your password into a computer you don't control. Now it's gone. Same for plugging in the USB stick. The Yubikey approach mitigates that.
Assuming you want to do this, the best practice you can achieve is just making the password long.
I mean, the password to the only ssh thing accessible from outside is 17 characters, and root is not ssh-able, only my user with a custom username