I agree that both matter, but one is more important than the other.

If they released the binary as "Open Source" but had a long list of things I wasn't allowed to do with it, the fact that they didn't release the source code would be of secondary concern to the fact that they're calling it "Open" while it actually has a trail of legal landmines waiting to bite anyone who tries to use it as free software.

And that's with a clear cut case like a binary-only release. With an LLM there's a lot of room for debate about what counts as the preferred form for making modifications to the work (or heck, what even counts as the work). That question is wide open for debate, and it's not worth having that debate when there's a far more egregious problem with their usage.