> There will be a huge market for local inference once it's cheap and widely available.

I've seen public pronouncements that the RAM shortage could persist for a decade.

And then if consider that the constraint on local LLMs isn't just memory size but bandwidth ...

If you take something like a DGX Spark and increase its memory to 512GB that doesn't even solve the problem. Because the bandwidth of DDR5 just can't manage reasonable speeds for decode. If you take a dense model or an MoE model uses up most of that 128GB in active decode you will only get like 15 tok/sec. "Real" datacentre inference boxes use high bandwidth memory that is 10x the speed.

I think we're unfortunately a long way off, unless people learn to accept working with much less intelligent models locally.

The innovation is going to have to come on the research & software side -- we need to find ways to pack more intelligence into a smaller number of parameters.