The place between my house and all of the other places I might need to go? Is your argument that people should just not go to the doctor or get driver's licenses unless they're willing to interact with every stranger who wants to talk to them lest they be "rude"? I don't understand how you can in good faith claim that needing to do something obligatory outside of their house or apartment is actively opting into social interaction with literally everyone else in public.
You can simply decline the conversation. It's not as big of a deal as you are making it out to be. I'm not saying you have to actively opt into social interaction, whatever that means. I'm saying its reasonable for others to offer social interaction if you're in public.