> This is an excellent use case for completely local, small model inference
I remember using a small model for autocomplete, I think it was Qwen 2.5 Coder 3B way back, on a laptop with 32 GB of RAM and a 13th Gen Intel i7 CPU. It made the fans spin up and loaded things so much (Ollama back then) that it was largely unusable for doing software development at the same time. I don't want inference anywhere near my personal computing devices (same in regards to phones, really, I don't want the power usage that'll just help degrade the battery faster). Also most small models just really suck, unless you have trained them on a very specific task, but then they suck at everything that is not that task.
Then again sending browser history online also sucks, unless you're into that kind of thing.
Not everyone uses laptops. I can run Qwen 3.8 27B (which is a REALLY capable model) in the background coding for me while I'm simultaneously browsing the web and playing VALORANT without any performance impact, and that's on a 6 year old GPU.