Recompiling the whole kernel just to change drivers seems like a deal-breaker for wider adoption

Recompile (or at least relink) the kernel to change drivers (or even system configuration) is a bit of a blast from the past - in the 1960s thru 1980s it used to be a very common thing, it was called “system generation”. It was found in mainframe operating systems (e.g. OS/360, OS/VS1, OS/VS2, DOS/360); in CP/M; in Netware 2.x (3.x onwards dropped the need for it)

Most of these systems came with utilities to partially automate the process, some kind of config file to drive it, Netware 2.x even had TUI menuing apps (ELSGEN, NETGEN) to assist in it

Not just old stuff like that either. At least also all the SCO Xenix & Unix'es up to the technically current OSR5, OSR6 and Unixware. I don't know about other (commercial) unixes as much as SCO but given where they all come from I assume Solaris and most of the other commercial unix that still technically exist today have something at least somewhat similar.

The sys admin scripts would even relink just to merely change the ip address of the nic! (I no longer remember the details, but I think I eventually dug under the hood and figured out how you could edit a couple files and merely reboot without actually relinking a new kernel. But if you only followed the normal directions in the manual, you would use scoadmin and it would relink and reboot.) And this is not because SCO sux. Sure they did, but that was actually more or less normal and not part of why they sucked.

Change anything about which drives are connected to which scsi hosts on which scsi ids? fuggeddabouddit. Not only relink and reboot, but also pray and have a bootable floppy and a cheat sheet of boot: parameters ready.

Quite common on Linux early days.

Also the only approach for systems where people advocate for static linking everything, yet another reason why dynamic loading became a thing.

If this kernel ever gets big enough where this might matter, I'm sure they can change the design. Nothing is set in stone forever and for the foreseeable future it's unlikely to matter.

If there's enough demand for dynamic kernel modules they can be added later. That's not a feature that you have to build ypur whole kernel around from that start. Linux definitely didn't but it has it now so it's definitely that can revisited or even made an opt-in feature.

Why? It can be fully automated just like dynamic module download and loading are.

Incremental compilation means you don't have to recompile everything just compile the new driver as a library and relink the kernel and you're done. Keep the prior n number of working ones around in case the new one doesn't work.