Most people running local models would probably love to run larger models if only they had access to big enough hardware. I'm curious: to those of you running models locally, if there was a way to inference the model of your choice at a reasonable cost by effectively time-sharing a B300 rack through some privacy-protecting intermediary, would you consider that?
If there was a "Mullvad of GPU clouds", would that solve the privacy concerns?
runpod.io is essentially this. You can rent the hardware for cheap in small time slices. I do this whenever I need to do a lot of embeddings, fast. I have an agent skill that will estimate the optimum hardware to reserve for the time/price constraints of the job, and you can spin up temporary inference for cheap via their API as well.