I would argue that’s an asymmetric game. If your boss’s boss or higher already dislikes your boss, then convincing leadership would be easy. But bad bosses don’t become the big bad boss by winning over their subordinates. They spend more time with their own boss than you do, your boss is likely going to have a better relationship with their own boss than you do, simply by proximity. Assuming they aren’t just a flat asshole of course.

A vote would in turn provide a mechanism to prevent such abuse. The boss works for the people not for their boss. Likewise the boss is no longer bound to the whims of their own boss. Their boss is the people below them. A group they will inevitably rejoin with new found appreciation for the troubles of management.