On my M1 Mac Pro I get 0.13s wall, so not much faster than your Mac. On my i9-9900K Linux box I get 0.04s. I would think my M1 single core performance would be on par, if not faster. Perhaps it has something to do with macOS and gatekeeper, as I notice I'm not getting as high of a CPU utilization.
$ gtime /opt/homebrew/bin/emacs --batch --eval '(princ (format "%s\n" emacs-version))'
30.2
0.07user 0.03system 0:00.13elapsed 78%CPU (0avgtext+0avgdata 46064maxresident)k
$ /usr/bin/time ~/bin/emacs --batch -eval '(princ (format "%s\n" emacs-version))'
30.2
0.02user 0.01system 0:00.04elapsed 95%CPU (0avgtext+0avgdata 57728maxresident)k
GUI Emacs on a 12 year old processor (i5-4590) feels faster than on a M4 Pro Macbook. I think it's just something to do with the window manager on each of the systems (my experience is mostly with Wayland KDE) rather than the speed of the CPU.
I also run GUI Emacs on both Linux and macOS. I build it on Linux with --with-x-toolkit=lucid and for $REASONS I'm still on X11. I run it in a full-screen frame on its own monitor, and it does indeed feel faster.