Any problems are inherent in the left-right metaphor.

To speed-run your second paragraph: (1) Absolutely the median person for any given question. (2) Suggesting that there's anything "neutral" about a median position is to catastrophically mix metaphors. They aren't remotely synonymous. (3) The failure of a highly partisan person to acknowledge gradations doesn't mean they don't exist.