Building a ramjet that is more efficient than rockets while travelling at several times the speed of sound is the easy part. The hard part is getting up to that speed first, because they give basically no thrust at low speeds. So you always need a two component engine and fuel system, with each component being useless at either launch or in-flight. Basically a similar reason why only the military uses VTOL planes. The military has no problem strapping a rocket booster on a ramjet missile.
And once you do all that, you will need to also handle the massive atmospheric heating from friction, so that the whole thing does not melt during flight.
Again, not that problematic for missiles due to shorter flight times and single-use ablative heatshields being viable.