On my 4090 setup I'm getting 86t/s on a 12B Q6_K quantized model running entirely in VRAM.
The current GPUs are optimized for processing huge numbers of triangles per second. There are three things at play here:
1. the organization of the data being sent to the GPU to optimize throughput;
2. the speed at which the GPU can read that data from its VRAM;
3. how many triangles it can process in parallel by using individual compute units.
I suspect that given parallel improvements for neural networks, we'll see similar improvements:
1. optimizing the structure of the weights in the model for efficient access by the CPU/GPU/NPU/TPU;
2. efficient access of data strides (matrix rows) in the memory, e.g. being able to read multiple 2x2 matrix values in one clock cycle, or stepwise pairs of values (a(i,j), b(j,k)) needed for matrix multiplication;
3. parallel compute for matrix and tensor multiplication and other operations needed by neural networks.
If you've got X GB of weights in slow access memory (be it RAM vs VRAM or SSD vs RAM) and Y GB of fast memory then no matter what, if you want to use them you'll need to transfer X-Y GB and will be bound by memory throughput. You can try to reduce number of activated weights, but how much can be gained that way is speculative so far.