The whole point is that you can't 100% trust the LLM to infer your intent with accuracy from lossy natural language.
Then it seems like the only workable solution from your perspective is a solo member team working on a product they came up with. Because as soon as there's more than one person on something, they have to use "lossy natural language" to communicate it between themselves.
Coworkers are absolutely an ongoing point of friction everywhere :)
On the plus side, IMO nonverbal cues make it way easier to tell when a human doesn't understand things than an agent.