Performance of WASM issues. Rendering performance of large data grids is not good. Also the first load time is also terrible 50mb+ payloads.
Blazor server uses websockets and is just a whole other bag of hurt. You'll have to deal with disconnects even if you can stomache the increased cloud costs.
50mb payloads are very extreme. There's definitely a few MB of hurt for WASM - but nextjs is just the same, plus it doesn't have the excuse of having to download the entire CLR!
You can (and I have) definitely rendered huge data grids efficiently with Blazor.
The biggest drawback with wasm is no proper multithreading support which has been delayed for years.
On blazor server; I totally agree, it's a pain. But for 'intranet' style apps which are used internally it's by far the most productive development environment I've used for web. I wouldn't use it for anything that wasn't a total MVP for public use but it's pretty great for internal apps like admin panels.