I have this exact problem, my colleague does not. I also noticed yawning helped. I gave my old AirPods to my ex :( I miss them.