Something about this is deeply funny to me:
> In an iterated prisoner's dilemma game with communication, agents all settle upon the same strategy and they all defect at the same time, tanking their overall rewards.
It’s not always consistent, but humans have a higher capability of self-awareness. It’s kind of telling that these Claudes don’t seem to consider this pretty obvious failure mode.
Overall I think this all makes me appreciate humanity a little more. Sometimes the truculent dev who stubbornly refuses to go with the flow produces very valuable insights, as a small example, discovering things the status quo thought unlikely.
I agree - I think one of the biggest reasons memory systems fail in LLMs is that they have poor theory of mind - they're terrible at considering how others will react. Both humans yes, but also future versions of itself. When asked to give advice to itself, it pontificates at length about trivial stuff it already knows and fails to emphasize the stuff that was new or interesting