Neat, thanks for the writeup! I think a single creator-admin for small groups is a nice, simple, and practical design point.
I did want to point out that Matrix does do distributed eventually-consistent authorization, which is their key invention IMHO. (Rooms are distributed among the homeservers, none of which are privileged over the others. You could (and their long-term plan from back in the day) was to run a tiny little single-device homeserver on every device to achieve P2P.)
It's tricky, but a very cool algorithm! Several entities (including myself as a hobby project) are working in combining the Matrix eventually-consistent CRDT with MLS for encryption for a no-compromise distributed E2EE system. It's possible, but very hard, as you might imagine.
Edit: Here's one academic paper writing up the abstract algorithm behind Matrix https://dl.acm.org/doi/10.1145/3381991.3395399
This is genuinely cool (and weird that I haven't heard of it). I released the 1.0 version today, but I'm already thinking about improvements for v2. Hopefully you will figure it out and I can implement it for v2 haha
Best of luck!