Not OP but listen to podcasts at highly accelerated settings:
The information density of ‘two dudes talking’ or any unscripted format is very low, so it time-compresses well. Specific podcasts, typically scripted monologues with technical content, such as Causality [0] (recommended!), I need to listen to much slower. Ditto if it is in an accent which isn’t mine, which slows my comprehension. I also slow the speed if I’m driving. So, yes, it takes mental overhead, but is doable. Go one click at a time and it will feel natural.
I suppose the format is a huge differentiator. I exclusively listen to highly produced content which has essentially no dead time. The content is already a compressed transmission of information.