On the other hand setting up the turn servers and so on for signaling before the webrtc connection is established can be a nightmare, if not using a third party service like twilio

Yes definitely. It's much simpler with a hosted service. Cloudflare offers one as well[0].

[0] https://developers.cloudflare.com/realtime/turn