I do not think you should ever feel that way. If any English-speaking listener has an issue with another speaking the dialect of English they were raised with, the listener has an issue with themselves they need to work through.

As an American English speaker, I have in the past used UK spellings when communicating at work with a group that I know only contains British English recipients. There is nothing wrong with that -- anything that makes communication more fluid should be welcome.

I believe the arrogance angle exists in a situation where an American English speaker with no British English education is using British spellings when communicating with other _American English_ speakers to purposefully create an air of superiority. If you do this, even if no one says anything, they definitely notice.

For other English dialects, my personal take is that most Americans (at least the ones who travel or interact with foreigners personally or at work) will assume they either are or are heavily influenced by British English due to history.