> 8 X RTX6000. It will run you around 80-100k to get started
8 x RTX6000 GPUs cost $100,000 alone. You then need to build a system that can support those GPUs with enough PCIe lanes through a PCIe switch.
It's going to be $120K to $150K to build or buy a system to run this.
Not to mention the three separate dedicated 15A circuits you would need to have installed in order to run the 3x 2000W power supplies running ideally at no more than 1400W sustained load each. And definitely would need 200A service to the house if you have a family living there with you.
But hey you could save on heating?
That’s a uniquely US issue - in NZ you can get a 100A single phase at 230V nominal without any issue. 23kw, straight to your door.
A single circuit using 10mm TPS would technically be enough to run what you’re describing. Might be pricey though, I’d probably take the excuse to get 3 phase installed so I could get access to the stock of used 3 phase machinery.
> That’s a uniquely US issue - in NZ you can get a 100A single phase at 230V nominal without any issue. 23kw, straight to your door
In the US it's common to get 200A 120/240V split-phase service. We're talking about the wiring inside the house, though.
How do you think everyone here is charging their electric cars at home and running our AC and electric cooktops at the same time if we didn't also have that? :)
You need to derate for constant loads here, and I assume you have to do that in NZ as well.
So, no, not a "uniquely US issue".
Not so sure about that. 200amp @ 240v is pretty standard for modern houses in the US. My house in Japan was only 40amps, so there are plenty of countries where this would be an issue.
isn't throwing that into a [insert financial vehicle that gives 99.99999% safe returns] going to destroy that when you factor in electricity costs?
Or even just electricity costs vs token cost
You can run the NV4FP quant with 8x RTX6000 cards at 50-75 tps output, but not (practically speaking) the OEM FP8 version. You will learn more about PCIe than you ever wanted to know.
The real gangstas are running 16x RTX6000s. Too rich for my blood, and the NV4FP quant doesn't seem to be that much worse.
Anyone done any benchmarks on the NV4FP quant? Seriously considering pitching an 8 x RTX 6000 Pro box at work to run GLM-5.2 in an air gapped environment.
At that price point you could also go with a Tenstorrent Galaxy Blackhole, which starts at $110,000.
Ooh, I hadn't seen these yet! That looks quite compelling, my only hesitancy would be what the software support looks like. But 1 TB of memory for $110k is really intriguing - I might go bother a sales rep. Thanks!
Good luck. I’m in the legal field, and even there, selling airgapped is tough.
What are the challenges you've seen in selling air gapped? Is it the high upfront cost? Challenges with hardware maintenance or something else?
We already use AWS. Everyone else is using AWS, so if there's an issue we can just say we were following industry standards.
My issue is we likely can't use AWS (non-US, CLOUD Act concerns + export control concerns).