> Wireguard can't punch through NATs or firewalls without third party software like Tailscale.

That's a false or incorrect statement, I've been using Wireguard and a cheap VPS (actually free on OCI) for several years, and with a cheap VPS at AWS Lightsail before that. No third party software in use at all. The only thing running on the VPS is Wireguard. The only thing running on my peers is Wireguard.

> Also I'm pretty sure each peer to peer connection needs to be individually set up in a config file ahead of time

That's how I do it but there are tools available to make it easy.