> but if all matter stopped moving, even in an isolated area, there is seemingly no longer a passage of time there

This is kinda what would happen in Penrose's Conformal Cyclic Cosmology[1], as far as I can gather. In his theory, he posits that in the far future, matter has decayed to radiation, and energy has been redshifted into infinity thanks to the expansion of space.

As such, in a local region things would seemingly be frozen in time. Nothing would change, entropy would not increase, and hence no apparent passing of time.

That said, his theory is pretty speculative. But fun to think about.

[1]: https://en.wikipedia.org/wiki/Conformal_cyclic_cosmology