Is a container not enough isolation? I do SSH to the host (alt-port) and then services in containers (mail, http)

Depends on your risk tolerance.

I personally wouldn't trust a machine if a container was exploited on it, you don't know if there were any successful container escapes, kernel exploits, etc. Even if they escaped with user permissions, that can fill your box with boobytraps if they have container-granted capabilities.

I'd just prefer to nuke the VPS entirely and start over than worry if the server and the rest of my services are okay.

Yea I feel that too.

there are some well respected compute providers as well which you can use and for very low amount, you can sort of offload this worry to someone else.

That being said, VM themselves are good enough security box too. I consider running VM's even on your home server with public facing strategies usually allowable

I understand where you are coming from but no, containers aren't enough isolation.

If you are running some public service, it might have bugs and of course we see some RCE issues as well or there can be some misconfig and containers by default dont provide enough security if an hacker tries to break in. Containers aren't secure in that sense.

Virtual machines are the intended use case for that. But they can be full of friction at time.

If you want something of a middle compromise, I can't recommend incus enough. https://linuxcontainers.org/incus/

It allows you to setup vm's as containers and even provides a web ui and provides the amount of isolation that you can trust (usually) everything on.

I'd say to not take chances with your home server because that server can be inside your firewall and can infect on a worst case scenario other devices but virtualization with things like incus or proxmox (another well respected tool) are the safest and provide isolation that you can trust with. I highly recommend that you should take a look at it if you deploy public serving services.