But, if I understand this correctly, if we make it speak "human", then we lose the accuracy of the results, and we're back in hallucination land.

I mean, maybe that's all right for when it's producing code - code is not beautifully flowing prose in the first place.

I think we can make it do both, but it is much harder because we have no algorithm to validate an AI's internal mumblings against proper, human-understandable English.

Code is not prose, yes, but as others pointed out, it is certainly nice if code is maintainable by humans.