> I wonder if podman would be a speed downgrade here?

Doubtful. But for sure it would be a compatibility downgrade. When I tried podman-compose a couple years ago, I found it nowhere near usable, though I'm told it's improved since. If you want a better Docker for mac, you're looking for OrbStack.