Most colors are mixtures of wavelengths of light. Our eyes have sensors that are sensitive to specific ranges of wavelengths, with relatively broad overlapping response curves. Orange can refer to a spectral color, but (as you say) the arbitrary nature of the names means it also refers to several mixtures of spectral colors of varying wavelength and intensity. "Brown" doesn't refer to any spectral color, it's always a mixture which could be referred to as a "dark orange".