Most more professional environments use VPN and bastion hosts for this purpose.

SSH is a risk because you’re trusting the users and client to not be idiots or get compromised. In general, people tend to do stupid things. If it’s just you and your server, potentially a different story.

It’s like sharing secrets with people, the more who are involved, the less likely the secret will be kept.