Have you tried scrcpy? https://github.com/Genymobile/scrcpy

I ran this with a custom fork to expose the device screen as a VNC server for years, no problems

Thanks for reminding me about scrcpy. I knew it but I've completely forgot about it. Will give it a shot. Why the custom fork? You added the VNC support to scrpy?

I had this old patchset: https://github.com/DavidVentura/scrcpy/tree/dev

Which added vnc support directly onto scrcpy, so I could leave a tablet plugged in to my headless server and access the tablet remotely via VNC

The alternative was to use X11vnc but it was janky and had some issues, plus higher cpu usage