You say it requires a little acclimatisation, but the caller may well just not know anyone with that specific accent/have never had the chance to acclimatise to it other than this phone call, there's an awful lot of accents in the world.
Additionally I find even though I can understand an Indian accent for example quite well in person, I really struggle with it over the phone due to the compression causing quite poor sound quality and lack of facial expressions to be able to read (which I would be using in person to help me understand a strong unfamiliar accent), whereas when accent is more familiar to me, the poor audio quality and lack of body language isn't nearly as much of an issue, presumably as I just have way way more exposure to the accent so can fill in gaps better.
You don't need to accustom yourself to every individual accent; you need to practice hearing people say things in different ways. Once you are comfortable with some voices that are different to yours, it's much easier to understand other differences as well.
I'm sympathetic to audio quality issues. No one would object if they developed tech to improve call quality, but they didn't.
I think you’re not taking into account that there are people from hundreds of countries out there whose native language is not English, that are not exposed to different English accents at all other than watching movies etc, and suddenly having to deal with weird accents.
You expect everyone to learn every accent?