I just installed a $16k GPU at home - super pricy to be sure, but when you price it against hiring 4 junior developers working around the clock it's a steal
My use-case is coding. I'm typically coding interactively with an agent most of the day, and then I have overnight jobs that run every night.
As for the infra setup, I am currently using llama.cpp, although I'm just now working on switching to vllm since it is better for concurrent inference, since it can interleave prompt-processing and decode. Until now llama.cpp was enough, since I was on an R9700 and it was only practical to run one agent with decent sized context anyway.
I just installed a $16k GPU at home - super pricy to be sure, but when you price it against hiring 4 junior developers working around the clock it's a steal
What if you compare it to an OpenRouter account and $0.50 per million tokens?
Oh that's way cheaper to be sure, but a lot less fun!
you also don't have to worry about hiding API keys or security of confidential data which is a bonus.
Could you explain what your infra set up looks like and what is your typical use case, if its more than a hobby?
My use-case is coding. I'm typically coding interactively with an agent most of the day, and then I have overnight jobs that run every night.
As for the infra setup, I am currently using llama.cpp, although I'm just now working on switching to vllm since it is better for concurrent inference, since it can interleave prompt-processing and decode. Until now llama.cpp was enough, since I was on an R9700 and it was only practical to run one agent with decent sized context anyway.