Traps were/are the mechanism for doing syscalls.

I don't know OG x86 (cuz, ewww) but on 68k this was generally the way. On my Atari ST a syscall was performed by filling your registers and stack as expected, then executing one of the TRAP opcodes and that would get the CPU To save PC etc & jump to the handler but in supervisor mode, where your syscall could then read state perform accordingly, and then return back to you.

I think x86_64 has just formalized this into a specific SYSCALL instruction?

ARM variants call it SVC (supervisor call).

Same difference.

Some older operating systems just implemented their syscalls as ordinary subroutine jumps, though, and everything ran in supervisor etc. I believe AmigaOS was like this, you just went through a jump table. Which, I think, shaves some cycles but also means compromises in terms of building for memory protection, etc.