My suspicion is that this is simply due to the M5 having more memory, and the OS already having most of the file cached. The M2 has more memory pressure and would cache fewer of the SSD reads
If that's true, inference speed would be even lower if you have only 2GB total, including OS caches
The base level M5 doesn't just have more memory than the base level M2.
The memory bandwidth is bumped up by 50%, and the size of the on-die system level cache is bumped up by 50% as well.