i am still confused what their software stack is, they dont use ceph but bought netapp, so they use nfs?

The NetApps are just disk shelves, can plug it into a SAS controller and use whatever software stack you please.

but they have multiple head nodes, so its some distributed setup or just active/passive type thing?

We have a custom barebones solution that uses a hashring to route the files!

I'm guessing the client software (outside the dc) is responsible for enumerating all the nodes which all get their own IP.

I think each rack is one head node and several disk shelves (10?). No dual headed shelves.