How does one self host an email server these days and not get flagged into oblivion?

At least use a service with actual customer support like Fastmail, and backup your email with an email client that downloads copies of your email to your computer.

I mean Proton seems like a good (although extremely expensive) option, but these small companies could go out of business on a dime which is a problem when the effort to switch is a titanic undertaking.

If you use your own domain, it's just as simple as switching some MX records (and the TXT or whatever for DMARC etc)