Did you try the NVidia driver from the Debian repo and/or NVidia themselves? The former would not load on my Optimus machines either, without any clue as to why. (I sunk more time than I should've into this. I literally tried everything on the wiki pages to discover they are horribly outdated.) I totally agree: this is not great. I know people will blame me for choosing NVidia, or NVidia, but why can Fedora, Ubuntu et al do this right?
Debian repo proper worked fine, but I hit a problem with an application that wasn't playing nice (Zed would just stop accepting text input...) with X11/Gnome and Wayland is a bad time on the 535/550 Nvidia drivers.
Realistically I can live on X11 outside of my dual monitor setup and that one application, but things get very choppy with mixed refresh rates. Still not the biggest fan of Gnome, but if I'm using deskflow only Gnome or KDE support the input sharing portal on Wayland.
The NVidia repo gives you 580, perhaps it is of help (I assume Wayland support is getting better all the time, if ever so slightly). I use KDE, so maybe that happened to contribute to a different experience.