When I started dating someone whose second language was English, I found that the idioms were a giant hurdle. We really do speak in idioms and metaphors and this makes it very difficult for people who understand the words but not the cultural context. While my (now spouse) adapted, I also adapted by learning how to speak far more literally, especially around people of mixed levels of English.

I don't think there is a fear of being imprecise. If anything, it is a fear of speaking plainly as boring.