A key difference between humans and bots is that it's actually quite costly to delete a human and spin up a new one. (Stalin and others have shown that deleting humans is tragically easy, but humanity still hasn't had any success at optimizing the workflow to spin up new ones.)

This means that society tacitly assumes that any actor will place a significant value on trust and their reputation. Once they burn it, it's very hard to get it back. Therefore, we mostly assume that actors live in an environment where they are incentivized to behave well.

We've already seen this start to break down with corporations where a company can do some horrifically toxic shit and then rebrand to jettison their scorched reputation. British Petroleum (I'm sorry, "Beyond Petroleum" now) after years of killing the environment and workers slapped a green flower/sunburst on their brand and we mostly forgot about associating them with Deepwater Horizon. Accenture is definitely not the company that enabled Enron. Definitely not.

AI agents will accelerate this 1000x. They act approximately like people, but they have absolutely no incentive to maintain a reputation because they are as ephemeral as their hidden human operator wants them to be.

Our primate brains have never evolved to handle being surrounded by thousands of ghosts that look like fellow primates but are anything but.