Permission guards solve one important problem: should this action be allowed?
The complementary problem is recovery. I run 8 agents with fairly hard boundaries between them, and I still hit failures where every individual action was allowed but the system broke anyway because two agents wrote shared state at the same time.
What saved that setup was supervision, not permissions. The memory server crashed, restarted cleanly, ran repair on boot, and the rest of the system kept moving. Permission checks stop known-bad actions; supervision is what makes unknown-bad outcomes survivable.