My experience mirrors that, but mostly because Caddy does HTTP/2 and http3 by default, natively. HTTP 1.1 is faster for bulk data transfers (while being worse at pipelining and latency in general), so I'm not surprised Caddy is taking more CPU in this case.

In my experience, in terms of latency, Caddy is a lot faster, every single time. I don't know what modifications I need to do to nginx to make it comparative but Caddy easily shaves half of the connection and transfer delays on my local network.