But then you weren't paying attention.

You context switch back and take 2 minutes (at best... much longer than if you were in the actual conversation paying attention) to answer. Now everyone else in the realtime conversation is waiting on your answer because you were expected to be in the conversation as well. That doesn't seem like good UX to me.