But if you eat food, I don't risk choking. They want us to take the risk for their reward.

But if I drive a car, You do run the risk of getting ran over. We can come up with any number of analogies of varying rightness and wrongness here.

And then there's a whole truckload of case law about liability that comes into play.

We haven't yet written those laws for "AI."

What do you have in mind?

You're paying for their services to collect reward for yourself, but also deciding your own risk/reward when choosing e.g. how much access to grant Claude for any given task.

I guess there's the case where the more capable Claude is, the more someone else can use it to find vulns in your services while Anthropic collects their subscription money? But that is mitigable risk that you shipped regardless of what Anthropic is doing.