If you're going to go all the way to the mouse, you could just... click the window you want.