Claude voice mode has come a long way! I'd say it's smarter than CGPT AVM last time i tried it.
But personally I've settled on just speaking to the slower models over a custom tts app, I find it being instant was not actually that important, and in the silence I find myself marinating in the discussion more anyway