But they could look up your physical address and store and sell that easily as storing and selling your delivery address now.

The third-party seller never gets your address. They hand the package over to the delivery company with a code, the delivery company is the only one who knows what that code means. Japanese delivery companies already offer this service, since a lot of people don't like to publish their address.