To me the value of the ESP32 has always been the peripherals that substitute for pure CPU usage (although, it's certainly nice that it has the CPU cycles to do interesting things).

You feed in the PCM samples, the I2S peripheral handles the PDM in hardware. So no CPU should be used.

Read the comment I'm replying to.