>We were building 40 Gbps packet sniffers at Google (4x 10 Gbps NICs) and needed switches that could do things like mirror traffic across ports at line rate
I'm sure there were good reaasons for this, but man this just sounds bad. Like that's the spec I think NSA must give to their contractors outfitting 611 Folsom Street's Room 641A
These tools are very common in non-nefarious ways for troubleshooting networking, and while vendors set up to serve this space solely as troubleshooting tools, I've also built my own that can sit on a network link and monitor for problems.
In my case it was for mobile wireless signaling traffic (all the coordination for creating a mobile internet connection, handing the connection off between towers, etc), and I'd credit it as one of the reasons you're mobile internet connection is so stable. When LTE first came out, myself and many others solved all sorts of bugs in the equipment and protocols by using or building these sorts of tools.
Port mirroring is the network engineer's equivalent of using breakpoints in a debugger for a programmer. It allows you to inspect what happens on the wire to get a clue for what's wrong.
It does sound bad the way OP described it, but packet or HTTP mirroring is a standard feature of A/B testing and blue-green deployments of a very critical code. Mirror traffic between version 1 and 2, then compare response body, response time and return response A to the end users.
But the comment says ”we were building packet sniffers” … and needed mirroring as part of that.
I used to operate several dozen 10G/25G/40G/100G taps which fed a series of tools in the DC at a regional healthcare back in the day.
80% of the relevant outcome was typically to get a true-to-the-wire sniffer capture (switch mirrors won't always mirror 100% of packets for various reasons) for troubleshooting performance of whatever the complaint of the day from the server team was.
19% was for feeding a security monitoring systems which looked for abnormal flow patterns to let us know a server was compromised.
1% was for the call recording system compliance requirement for the emergency department.
0% was because I was a cool superspy tasked by the government to siphon info to them or trying to sell medical records on the black market or something. I mean, you can try to something nefarious with such tools... but one could say the same about a generic server, SAN, application, etc as well. People are just used to understanding what those would typically be used for so they don't assume it must be for the scary thing they've heard about.
That said, it doesn't rule it out either. But again, the concern shouldn't be sourcing from their usage of normal infrastructure tools it should be sourcing from... well, all of the user analytics Google very publicly does directly in the server.
I was under the impression that the superspy stuff usually happens more at the undersea-cable terminal or at satellite links anyway.
It's super common and very useful for security systems to work like this. Instead of putting something in-line that might choke on a burst of traffic you put it off to the side and send it a firehose of packets it may or may not be able to handle. If it falls over, no problem.
Common tools for any network engineer
Nah, packet mirroring is a standard networking feature and ideally every feature is line rate.
Indeed. Juniper had this over 20 years ago.
It came for free on hubs :)
the cost there was performance
There are companies that mirror their traffic to storage for a few days to be able to look back and troubleshoot issues. ALL their traffic, proactively.
For US federal government, "Office of Management and Budget" says to store 72 hours of packet capture, since 2021 (partially in response to SolarWinds attack). (M-21-31)
https://www.cisa.gov/sites/default/files/2023-02/TLP%20CLEAR...
I do that with very, very low traffic industrial automation systems.