Performance can be fine for the vast majority of hardware. Some drivers may need to be colocated in the same process for performance, but your average PCI driver doesn't benefit at all from being in the kernel. People also underestimate what you can accomplish with an efficient async first shared memory based ipc can accomplish.

Security benefits of driver's being in user space become limited quickly if you lack an iommu. Additionally if it has to set things like voltage regulators or clocks it can easily put the system into precarious states. That said it's still worthwhile and has lots of other benefits.