I've found the same thing with short walks without headphones. The first few minutes feel almost irritating, like my brain is looking for something to latch onto. Then after a while the mental noise settles and work feels less aversive again...