But notice your own option: "stop communicating with people you don't trust." For a group, kick is an option: everyone stops including X, at once.

And that's where the leader matters -> to make that option executable as a group. Without an agreed authority, it's N separate choices that have to stay consistent forever: one member with a stale roster keeps X in the loop by accident, or X kicks Y while Y kicks X and there are two rosters claiming to be the group. One signed kick says "we stopped talking to X" for everyone.

But then again, you're right. If 4 members want to kick out member#6, but member#5 doesn't want to, there is nothing we can do to stop member#5 from sending everything to member#6. That's not a software-solvable problem.