Agree with this. As soon as things get in range for motivated amateurs, progress skyrockets.

Has also been the case for things like chess computing; a lot of the progress we made over the last decades there (even before involving neural networks!) happened thanks to software improvements because the problem got so accessible, not just faster hardware.

I expect similar trends with AI; I'd expect to get decent, human comparable capability with <200GB/s of memory bandwidth and under 60GB of RAM long term (SSDs with very high read bandwidth looks also promising, but we'll see).

I hope that in a decade or two, training will also be somewhat feasible for "pro-sumers".