Thank you so much!
Most daily commands like launching and moving around sites/apps happen without AI, but I agree with the privacy concern and will definitely look into utilizing local AI.
Smaller models, like Haiku, I have found to already struggle during multi-step tasks, so I expect local model capabilities to reduce performance further.
I'll look into how OOMU splits their work up, but what local model would you suggest starting with?
The choice of model could be up to the user, depending on the available RAM on their M class Mac. Newer and bigger usually being smarter.
Llama.app/.cpp and Unsloth Studio on Mac already provide guidance to pick the optimal model for the available memory, all model wheights hosted by huggingface. Both apps can use the local huggingface model cache locally.
If you make it configurable, the user could also run a separate machine with an NVidia card and Linux to serve tokens on the LAN. Or use OpenRouter or any other cloud hosted model offering (DeepSeek, Z.ai, etc.).
I would probably run qwen 27B or qwen fn but I would run the actual inference on one of my other boxes and not on actual desktop so initially it could be sufficient if you just enabled openai or anthropic compatible endpoint.