> I'd figured that UNIX time just counts actual seconds

It doesn't. UNIX time is 1:1 with what your clock reads, you can convert between 2026-07-10 00:12:25 UTC and 1783642345 by simple math (note how the last digit is the same; but the leap second offset is 37s, that wouldn't align. The TAI timestamp for the same moment in time is 1783642382.)

> and that leap seconds and similar calendar shenanigans would be a problem of mapping epoch to the correct date, so that if normally epoch X maps to date D then both epoch X and X+1 map to date D.

It's the other way around, if you need to get the exact seconds between two UNIX timestamps (or calendar date/time), you need to check if there were any leap second changes between the two.

Counting actual seconds is TAI [https://en.wikipedia.org/wiki/International_Atomic_Time], that one is really just "number of Caesium hyperfine transitions since X", and now the mapping to a calendar date/time for display is a bit involved.