Apparently the color name did come from the fruit, and therefore didn’t enter common use in English until around the 16th century.