It would be awful if running models locally became the primary way of using LLMs. On dedicated servers sharing GPUs across requests, energy usage and environmental impact is way lower overall than if everyone and their mother suddenly needs beefy GPUs. It’s the equivalent of everyone commuting alone in their own car instead of a train picking up hundreds at once.
You can batch requests when running locally too, if you're using a model with low-enough requirements for KV-cache; essentially targeting the same resource efficiencies that the big providers rely on. This is useful since it gives you more compute throughput "for free" during decode, even when running on very limited hardware.
That’s still orders of magnitude less efficient, and also not how most people use AI, or probably will use AI.
It's even more awful if the compute capital is owned by only a handful of players.
Maybe people would target their use more appropriately, then.
Just like people would drive their car as little as possible out of concern for the environment..?