I'm talking about Gemini Live in the Gemini App, not Google Translate.

Tell Gemini Live which speaker you want to engage with, and it is relatively intelligent about it.