this sounds like something super weird about the setup - it really should just work. what client and server was it?

Element web and mobile have the same issue; server I'm not sure, where can I see this? I would assume the default/reference implementation but I don't know

Groups work btw, it's just private chats that cannot be initiated from either side (I have like five pending 'rooms' or something now), so it's not that one server banned the other. I just took it for typical matrix experience