Rise of local models? what rise? I run qwen3.8 flash next on my 200GB/s 128gb Strix Halo and even that is barely acceptable, big models require so much more. Nobody's running a local model on their power sipping phone. It's just a nerd's wet dream.
Rise of local models? what rise? I run qwen3.8 flash next on my 200GB/s 128gb Strix Halo and even that is barely acceptable, big models require so much more. Nobody's running a local model on their power sipping phone. It's just a nerd's wet dream.
That's true today, but what will it look like in 20 years? Over time, algorithms and hardware always gets faster and more efficient. Like if I wanted to run a fluid dynamics simulation, what would have required a supercomputer 20 years ago can now run on a single desktop computer.
256x speedup on FAT32 just earlier today; I expect we'll have a nice bump in performance from domain cross-pollination for a good while now.
barely acceptable for all use cases? like a speed issue?
Regardless, looking back even a year at SOTA local models that fit on a non-absurd system creates a crazy trendline toward the future. And I think model routing is the gap filler that can make/break now vs. Oct 2027
I suspect Anthropic is of this general thinking as well; they've clearly switched from a thoughtful progression toward a natural IPO to "omg sprint for it". Along with the regulatory capture attempts, it's hard to imagine what is breathing down their neck so hard outside of this.
Totally agree local inference of any major scale is a nerd's wet dream short of many stacked improvements in efficiency. But again, auto-routing smooths over the rough edges if you are ok with a little bit of API spend when needed.