Do they care about locally-hosted, or only about self-hosted? I'm not really clear why a local box would be any better than running on a private AWS instance in any of these scenarios...
Do they care about locally-hosted, or only about self-hosted? I'm not really clear why a local box would be any better than running on a private AWS instance in any of these scenarios...
For one, doing the math on what it costs to rent a 768GB+ RAM AWS system with 40+ high performance CPU cores makes it very unappealing to pay for 12, 24, 36 months of it.
The largest high performance compute ec2 offering, the c9g.metal-48xl , maxes out at 384GB RAM and already costs a shitload.
The m9gd.48xlarge and m9gd.metal-48xl both have 768GB RAM and I cringe to think what they cost monthly. I just did the math on one of these and it costs $12 per hour, or $289 a day, or $8900+ for one month.
Also plenty of Europeans or people from other locations may consider it as an unacceptable risk factor to put their "off site" self hosted AI stuff with an American controlled company. Particularly if the servers are physically in the USA.
Hetzner will also rent you 768 GB of RAM with a Blackwell 6000 Max Q GPU for €2300/month [1].
Yes, it's a boatload of cash, but that's a €13,000 GPU and €20,000 of RAM at present prices. There is a segment of businesses where a fixed €28k/year bill is going to be preferred over plonking down €40k for a (theoretically) depreciating asset and ongoing colocation costs.
[1]: https://www.hetzner.com/dedicated-rootserver/gex131/
Renting something at a rate that'd be purchased in less than 2 years seems very myopic to me. And yeah it depreciates, but not to zero. So if you're speaking of the breakeven point after liquidation, you're probably there in well under a year at those rental prices.
> Renting something at a rate that'd be purchased in less than 2 years seems very myopic to me
And yet basically all AWS customers are doing exactly that. Turns out that making CAPEX "someone else's problem" is worth quite a lot to many businesses
that would be implying that "private" really means anything for AWS. Because if it's "private" as in "private" github repos that were totally not used for training copilot because they said so or "private" claude chats that are totally scanned even if you have enterprise contracts to check you are not doing anything malicious or are from china or whatever, and this will totally not be used for training...
can we trust any US based service to guarantee privacy and confidentiality? especially to us european frienemies?
> that would be implying that "private" really means anything for AWS
Insert your dedicated hosting provider of choice for 'AWS' (somewhere like Hetzner will be cheaper anyway).
But in general, AWS hosts are yours, running your code, with your security policies enforced. Sure, the US government can silently subpoena the contents thereof, but aside from that fairly extreme case, it's not like AWS is handing your data over to 3rd parties.