The logic here is a bit different. Consumer software is slow because it's the user paying for the compute to run those bloated Electron apps. Unless we're all running local inference in the future, it would be the vendor paying for the compute to run the AI-based UI.

Of course they would pass that along to the consumer, but nobody is going to pay token-based pricing for something that used to be free and ad-supported, or a cheap monthly subscription.

This demo is already using a local model. Which is why I made the Electron comparison.