There's definitely some physical underpinnings--most music systems have the concept of an octave which maps nicely onto frequency-doubling, for example. But there's also culture: For a purely Western example, even-tempering is in tune, but you'll hear different "beat patterns" for a given interval than with an instrument tuned for music in just one key.

And the choice of which ratios "sound good" is cultural, to some extent.