I'm interested in this - I am from the Commonwealth and I do use those words, including when I forget with American colleagues.
It never occurred to me that this could ever be perceived as arrogant (even if only when referring to someone with a different background.) And I wouldn't have thought it would mean anything more than a certain language cosmopolitanism, lah ;) (Hope that joke comes through! It's been decades since I had much exposure to Malaysian English.) Can you explain why this might be, please?
Speaking for myself (an American), when I read published work that uses British spellings and I know the author is American, it feels to me that the author thinks American spellings are somehow vulgar or improper and he/she is trying to rise above our shameful misspellings.
British Commonwealth authors (well, really any author I know to be not from the USA) get a pass because these are the spellings they were taught. Nothing wrong with that.
This is a phenomenon I've only noticed in the last two decades or so. I don't know if American students are now being (wrongly) taught British spellings in school or they merely think their writing will carry more weight if it has a British "accent" but it just seems arrogant to me.
The OED is a useful resource but it is not our dictionary of final arbitration. Americans should use the American Heritage Dictionary.
I do not think you should ever feel that way. If any English-speaking listener has an issue with another speaking the dialect of English they were raised with, the listener has an issue with themselves they need to work through.
As an American English speaker, I have in the past used UK spellings when communicating at work with a group that I know only contains British English recipients. There is nothing wrong with that -- anything that makes communication more fluid should be welcome.
I believe the arrogance angle exists in a situation where an American English speaker with no British English education is using British spellings when communicating with other _American English_ speakers to purposefully create an air of superiority. If you do this, even if no one says anything, they definitely notice.
For other English dialects, my personal take is that most Americans (at least the ones who travel or interact with foreigners personally or at work) will assume they either are or are heavily influenced by British English due to history.
I have always been "aware" of the concept that they can be perceived as arrogant, but really only "colour" - it sounds kind of deliberate and like some attempt to sound "fancy," like enunciating "theater" as "thee-AY-tour" But even so, I usually see it as a humorous thing. The person is purposely trying to sound over the top arrogant/refined as a joke. I've never actually read "british" spellings and gone "what an ass." I usually assume that's how they write or it's a joke.