> My gut feeling is that this is way harder than most people think
I've had this feeling for a while too; partially due to the screeching of "putting your ssh server on a random port isn't security!" over the years.
But I've had one on a random port running fail2ban and a variety of other defenses, and the # of _ATTEMPTS_ I've had on it in 15 years I can't even count on one hand, because that number is 0. (Granted the arguability of that's 1-hand countable or not.)
So yes this is a different thing, but there is always a difference between possible and probable, and sometimes that difference is large.
Yeah, you're getting fewer connection ATTEMPTS, but the number of successful connections you're getting is the same as everyone else, I think that's the point.