This should be obvious but with a model running on local hardware you can do your own RLHF and mod its behavior however you see fit. With cloud hosted models you can't. A few years ago when the models were smaller there were people undoing the guardrails, censorship, and general lobotomization with some form of a RLHF training. You can't do that on larger models unless you have the hardware like this person does.
Notice all the comments saying like "omg why so expensive so just use the API??". It's a trick for lockin even with, so called, "open" models. Keep trying to run them locally, keep undoing the lobotomies, mod model behavior so that they work for you and do what you want vs only what someone else says they're allowed to do.
I love my Spark-like, but even for training you're better off using Vast or Runpod or whatever to rent cloud compute. Much faster and cheap as hell, to be honest.
I do set up my initial runs and likes like quantisation-aware-distillation on my Spark-like to test it out and get it working, so it has value! But its not "worth" it other than its fun hardware to tinker with, IMO.
> You can't do that on larger models unless you have the hardware like this person does.
Or just rent something substantial for like $4/hr on runpod or w/e to do that.
My gripe is this persons compute is wasteful and makes it harder for me to buy something with like 64gb ram to do normal work and run containers while I keep using cloud models.
Someone else calculated the break even being 10 years, it’s just dumb. And I think it’s clear there won’t be a big rug pull anymore, there are too many open models and providers now.