I don't really get the ick about wearing these kinds of things in public, but, having tried to wear XR glasses in public has shown me that the bigger issue is that these things are at a whole other level of distracting compared even to phones. It'd probably be worse with a full face headset, since at least with glasses you have your full field of view.
Even despite pinning the display in a corner, I noticed I was very distracted when crossing streets. Made me stop wearing them like that, instead prefering to carry them in my backpack and use them on train rides or flights.
At least with my phone I can just slip it back in my pocket when I need to pay attention to my surroundings.
I don't think using a display when walking around is a good idea or even just safe to do. On the other hand, I think it makes perfect sense to have something a like a fighter jet HUD - important information in static places (time/location/notification status/etc.) with possibly some highlights (showing where is the next bus in AR, annotating things when you look at them, seamless translation of text, etc.).