Yes there are diminishing returns, but they're a long way from where you start.

Pre-industrial revolution most people in my country own enough clothing for everyday and then maybe some extra stuff for "best" appearance like church. Making a shirt requires hours of labour from skilled workers. The machines eliminate one of those jobs ("hand" weaving, using a loom that you operate at human scale, replaced by enormous machine looms which produce much more fabric and much faster with far less human labour) and streamline others so that the price of a garment collapses. So people buy a lot more garments. You probably own enough clothes that you can wear different clothes each day for at least a week, likely more, and you think nothing of it.

Most people won't buy so many clothes that they can't even wear items once, but many get pretty close, and so worldwide we produce hugely more clothing per person than before the textile mills were first built.