I don't know if you're in a minority. I think people just don't like a boring answer like "silence".

I was raised in a big family, and I prefer silence when I need truly deep focus. From my experience in open floorplan offices, a majority don't break out the headphones until it gets noisy enough. Some people would even come in early or stay late for exactly this reason.