Linux kernel compilation time depends heavily on what you're compiling though. You can have wildly different compilation times just by enabling or disabling some drivers or subsystems.

Yep - I have a 9950X desktop. Building the kernel as part of NixOS builds almost every possible option as a module - for ARM64 that takes something like 15 minutes.

Let's also not forget that kernel was not stagnant, but grew a lot over the years.

Also true! Hard to say which was moving faster, though: CPU speed increases & RAM amount increase, or the additional code and complexity written into the kernel.