For those not keeping count, total hardware spend is in the 13k-20k USD ballpark, by my count.
The thing that I like about this post is that it touches on many of the difficulties of running a homelab with many physical hosts. You might not need all or most of this setup, but at least you have an idea of how this particular design (a decent one!) scales after reading this.
- Array of off-the-shell compute servers with console access + networking + power
- ArgoCD + GitOps makes a K8 cluster declarative
- Talos makes the physical hosts that provide the K8 cluster be declarative
- Dedicated machines for storage, Control Plan, and Networking isolate the infrequently-changing stateful parts
This homelab does seem compute focused, which might be right for OP but is normally a mistake that people make when they build their first homelab. I'm wondering what OP's home internet bandwidth is. It seems odd to have so much compute behind a network bottleneck unless OP has high-compute-small-output workloads (ml training, data searching/analysis, NOT video encoding)
A single machine with a lot of CPU and a LOT of memory/storage is typically what people want---so that projects they're setting up are fast and having lots of old/idling projects is fine. My old recommendation was a mini-ITX with 128 GB of ram and a modern AMD cpu should take most people very far. Perhaps a smaller NUC/Beelink computer if you're not storage hungry.
However, keep in mind that a single machine will make it hard to tinker with large parts of the stack. It's harder to play with kernel mod settings if you need to constantly reboot all your projects and possibly nuke your lab. It's harder to test podman vs docker if in involves turning off all your contains. A more complicated home-lab gives you more surface area to tinker with. That's both more fun and makes you a better engineer. Of course, you can get most of this experience for far less money if you budget isn't quite as generous.
I personally prefer a digital nomad aesthetic, so I focus on small & simple on-prem hosts paired with cloud stacks. I'm willing to pay a premium on compute to have less commitment and physical footprint. I've been considering setting up a K8 cluster on Hetzner dedicated machines. In my case, that Mini-ITX box is actually a storage-optimized ATX build for backing up my laptop (daily-driver) and media server.
> This homelab does seem compute focused, which might be right for OP but is normally a mistake that people make when they build their first homelab.
I kept waiting for the description of what it would be used for, but there was only a passing reference to learning how to run AI workloads.
For some people, buying and assembling hardware is the hobby. This gets old fast, especially when you add up how much was spent on hardware that's now sitting idle while it becomes more outdated year over year.
I agree that for typical learning cases the best solution is a single, cheap consumer CPU paired with a lot of RAM. For the $8000 spent on those 8 mini PCs, you could build a 256GB RAM box with 2 or even 3 nVidia 5090 GPUs and be in a different league of performance. It's also much easier to resell big nVidia consumer GPUs and recoup some of your money for the next upgrade.
It does look fun to assemble all of this into a rack and make it all work together. However, it's an extremely expensive means to an end. If you just want to experiment with distributed systems you can pair 128GB of RAM with a 16-core consumer CPU and run dozens or even 100 small VMs without issue. If you want to do GPU work you can even use PCIe passthrough to assign GPUs to VMs.
> I kept waiting for the description of what it would be used for, but there was only a passing reference to learning how to run AI workloads.
Future posts will address some of this. :)
> For those not keeping count, total hardware spend is in the 13k-20k USD ballpark, by my count.
Yep! Right around the $13.5k mark.
> This homelab does seem compute focused, which might be right for OP but is normally a mistake that people make when they build their first homelab.
Very compute focused for the specific projects that I intend to work on.
> I'm wondering what OP's home internet bandwidth is. It seems odd to have so much compute behind a network bottleneck unless OP has high-compute-small-output workloads (ml training, data searching/analysis, NOT video encoding)
1gbps symmetric + a failover Starlink connection. Not a ton of large output workloads at the moment.
> However, keep in mind that a single machine will make it hard to tinker with large parts of the stack.
Very much in agreement here. This is one of the reasons I went with multiple machines.
> I'm willing to pay a premium on compute to have less commitment and physical footprint.
I also like this mindset, but my other hobbies are piano (and I'm very sensitive to the way that the keys are weighted, so I prefer playing a real grand piano vs a portable/mini electronic piano) and woodworking (even more bulky equipment), so I'm already pretty committed to putting down roots.
Sorry if this is a naive question, but with a bunch of power packed into a single device couldn't you do a lot of experimentation in VM?
The SER 9 workers in particular feel like they're overly beefy. I run 4-5VMs inside Proxmox in a single equivalent box...