Moore's law for 10-15 years was more like 20-100x ram sizes, not 2-4x.
Performance, storage, etc is definitely getting better, but it's a different scale of improvement
Moore's law for 10-15 years was more like 20-100x ram sizes, not 2-4x.
Performance, storage, etc is definitely getting better, but it's a different scale of improvement
10 to 15 years from now the scale of LLM efficiency improvements is, quite literally, unpredictable.
It could be that the company valuations crash tomorrow, and (almost) only performance gains achievable on hobbyist-level hardware come to fruition from there on out.
Or it could be that in the future, we have a custom "model FPGA" à la Taalas [0] in every home, and that it turns out we can still massively boost inference efficiency due to novel discoveries like TurboQuant [1] or a somehow-improved quantization method [2] again and again ten times over.
Point is, Moore's law in this context shouldn't be applied to just hardware spec sheets alone, but more the total number of "parameters potentially improving", IMO.
[0] https://chatjimmy.ai
[1] https://research.google/blog/turboquant-redefining-ai-effici...
[2] https://prismml.com/news/bonsai-27b