I used to have an SFP28 Mellanox card in my home server, but went back to a simple 2.5G Ethernet port for the LAN side. The Mellanox card ran hot and needed an extra fan near it to dissipate the heat. It was cool but there was no real benefit other than occasionally when transferring some large files.
Until motherboards include SFP ports it's probably not worth the effort at all in home setting; external adaptors like the one presented here are unreliable and add several ms of latency.
> Until motherboards include SFP ports […]
A micro-ATX motherboard with on-board 2xSFP28 (Intel E810):
* https://download-2.msi.com/archive/mnu_exe/server/D3052-data...
* https://www.techradar.com/pro/this-amd-motherboard-has-a-uni...
For reference, I'm seeing pings from my Mac to my Linux boxes (Lenovo Tiny5) at well under 1ms, not much worse than between them directly. But yeah, your mileage may vary.
Yep, these cards need a fan (or any kind of directed air flow).
Where did you get "several ms of latency" figure from? I have not measured external card, but may be I should do it... Because cards themselves have latency in range of microseconds, not millis.
I haven't tested this particular Thunderbolt SFP adapter, but my experience with a TP-Link 1Gbps USB adapter is that it adds about 4ms of latency. Far from being unusable and similar to WiFi perhaps, but worse than PCIe cards that should be <1ms.
it's all just driver/options crap if I were to take a guess.
there are a lot of usb options that matter, and tp-link ships lots of realtek chipsets that require very special driver incantations that a lot of the linux drivers simply don't replicate.
two+ layers of bad options will surely add 4ms quick.
I think there's definitely something with that specific setup. For me, pinging between two cheap Realtek 2.5 GbE USB dongles (one is on a Mac one is on a 7 year old Intel Atom Synology) is still sub-ms (hovering around 0.7-0.8ms) so it's not an inherent problem to USB dongles.
USB itself can have a lot of issues anywhere in the chain. I have a Thunderbolt dock where half of the USB ports adds latency and reduced throughput just because the USB chipset that powers them is terrible (it has two separate USB chipsets from different brands). Switch to a different port on the exact same dock and it's fine.