“100% guaranteed downtime when you least can afford it and the support tickets are your problem.”
We’ve a hybrid shop, including hosting our own ML infra, and we save a ton from cloud spend with local ML. Easily one million USD over past three years. But it’s not “free”, you are shifting a lot of labor into your plate.
And with that also gain institutional knowledge, skill up your workers and attract talent that wants to work on this stuff.
All boils down to short-term/long-term thinking.
This. People WANT to work on this stuff. And having skilled workers is a precious advantage.
Still has to break even on the balance sheet, especially at a bootstrapped startup. We actually made most of the financial windfall in translation API fees oddly enough.
For our own model training we needed to do some large scale translation tasks of a large dataset (1M or so documents, 10 or so target languages), running full-size NLLB on-prem saved us an absurd amount of money vs Google Translate API.
(For reference doing 1M target docs into a single language in Google Translate API is roughly $120k list price. You can run full size NLLB on an 48GB NVIDIA A600 and the major difference for us was speed, but for this task time to completion wasn’t an issue.)