What we really need is a breakthrough in inference or LLM architecture to allow running GLM-5.2-level models at the size of Qwen 3.6 27b or smaller on consumer devices like a 48GB Macbook Pro, and at least at 100 tokens/second. My hypothesis is that a smaller, less capable but faster model paired with a good harness can run for longer and brute force its way out to solve problems that the bigger models can one-shot.
That would be great during the winter months
used to heat my dorm room with a Pentium D that I overclocked and pointed a small fan at. Could open a window in winter, turn off the room radiator, and keep it cozy.
perhaps at-home LLMs will bring me back to that. fun days of hacking and thermodynamics.
Why not! And free hot showers during the summer. Just looking at the progress made since 2023, I don't think the LLM architecture we have today is the most efficient. We need creative game developers to start making LLMs.
Unfortunately I don't expect indie game devs can afford high spec hardware at current prices.
im more expecting the harness to be a literal LLM, Like how you put vibration dampeners on all kinds of mechanical structures