Looks incredibly cool, although it ran at ~4 tok/s on my browser (I assume because I've disabled V8). I believe your numbers though. I always wanted to try using something like this to create an AI assistant for my phone. If you try to create something without AI, you'll quickly run into the problem of using NLP to map input to the equivalent of AI tool calls. On the other hand, shelling out to a full-blown LLM for every input is slow, costly and if you want it local, greatly restricts its computing power and the number of devices it can run on. Needle and similar models solve this problem nicely: they're incredibly cheap and handle NLP (not in a predictable way of course, but with great success rates), so you can quickly run predefined actions or shell out to a proper LLM for more complicated tasks. You have the best of two worlds.

You may have inspired me to give it a try!