Thrift stores are .99 to 2.99. The local library has a for sale section with a massive selection of music. Ranging from .49 on up. Ebay is good for more rare albums but there's so many shops selling them also for dirt cheap free ship, to the point where I don't even know how they profit.
I've also had amazing luck going to estate sales and just asking if I can buy them all out. The last one I paid under 10 cents a cd. People just want to get rid of their 'old tech no one uses anymore'.
Most people realistically are listening to the same music multiple times over multiple months. If you listen to a different album every day you do kinda fall outside of the norm and ya, that would probably make streaming a better choice.