As you said, it's the set up. I'm almost exclusively using a single monitor, which works well when there's zero cost to switching desktops.

For your use case, imagine if the Window switcher, instead of being instantaneous, was a ~1200ms animation which blocked all key presses during the animation.