For a long time, the chip manufacturers had an inclination to simplify the hardware and rely on the software adapting and optimizing. But for decades this bid failed. Now we have the unrelenting AI capable of finetuning kernels relatively quickly. Maybe simpler hw will work this time? Note: not sure if TPU/NPU is not only simple but also too limited.
I have some doubts. For example, there is no software replacement for out-of-order execution: It is unknown before runtime in which cache level the required data will be or which values it will have (which changes the latency of a few instructions).
What if caching was not transparent, but required explicit management?
That was the PS3 SPU. It was very fast for its time, but only for the small subset of code that could work within its constraints, and viciously difficult to program for.