Drivers are given the latitude to kick passengers off public transportation for a range of reasons including violating the rules of conduct.

If I'm reading this correctly, you're talking about panhandling as an example of asshole activity that you believe is protected by the 1st amendment. Specifically, in NYC, it was ruled that panhandling is not protected speech on public transportation, see Young v New York City Transit Authority.

Of course, if the passenger refuses to leave or stop, the driver can't physically force them to and must escalate to a police force. Although, I imagine that's similar in many other countries as well.