It's incorrect to reduce polyphony to just shifting harmonies. Harmony is of course the constraint but the whole point is about intertwined voices - various melodies that your brain can process simultaneously. Music is not speech, but it's not that far from the language as it seems in terms of processing. When you hear 2 melodic lines at the same time, you're brain needs to encode them separately, in a similar manner to parallel speech streams.