For example, if you opt in, we'll tell you how you're doing compared to other users. The obvious AI variant is to let your users opt in to training your model on their interactions with it. Many will resist that, but if some don't, the model they get to use will outperform the vanilla one used by the others.

The vanilla one? Is this the pg Malcolm Gladwell arc, where he's retelling some real concept but warped by misunderstanding to the point of incoherence? Who's getting hobbled running on "vanilla models", do we have examples?

If it won't make the model worse when you opt out, then you have to find a way to make sure it makes the model worse, so they'll opt in.

It's like how LG TVs make you consent to recording and uploading everything you say in the vicinity, or else you can't have voice commands, which they explain is because it would be illegal to analyze sound inputs for the voice commands without that consent. It's not true, of course, but it sounds true and it makes you want to consent.