I think you're making good points, that aren't exactly counter-examples to the concerns being raised.
You are making the - correct! - point that _other companies_ who have paid contracts with an AI provider would impose significant costs on that provider if those contracts were found to be breached. Either the company would leave and stop paying their huge subscription, and/or the reputational fallout would be a cost.
But companies aren't people, and are treated differently from people. Companies have lawyers. Companies have the deep pockets to fight legal cases. Companies have publicity reach. If a company is mistreated, it has the resources and capabilities to fight back. A person does not. If J. Random Hacker somehow discovers that their data is being used for training (if they even could), what are they gonna do about it - stop paying $20/month, and post on HN? That's negligible.
So - yes, you're right that there are cold-hearted profit-motivated self-interested incentives for an AI provider to not breach contract to train on _a company's_ data. But there is no such incentive protecting people.
EDIT: /u/johnnyanmac said it better than me:
>> If they promise those paying customers (in legally binding agreements, no less) that they won't train on their data... and are then found to have trained on their data anyway, they wont just lose that customer - they'll lose thousands of others too.
> I sure wish they did. In reality, they get a class action, pay off some $100m to lawyers after making $100b, and the lawyers maybe give me $100 if I'm being VERY generous, while the company extracted $10,000+ of value out of me. And the captured market just keeps on keeping on.
Yes, my argument is mainly with respect to paying customers who are companies, not individuals.
I have trouble imagining why a company like Anthropic would go through the additional complexity of cheating their individual customers while not doing that to their corporate customers. That feels like a whole lot of extra work compared to just behaving properly.
Especially given that companies consist of individuals, so the last thing you want to do is breach the privacy of a personal account belonging to the person who makes purchasing decisions at a large company!
I mean this earnestly, not snidely - I wish I still had the faith that you do in not being treated abominably by any and every company, or to believe that they wouldn't default to behaving improprerly at any opportunity and for the barest profit margin. It would be nice to still believe that good things could happen under capitalism.
(as a sidenote, I'm very grateful for your insightful and balanced writing on AI in general. It played a considerable part in convincing me to give AI tooling another go after I'd initially written it off as more trouble than it was worth)