I can imagine, where this becomes a mainstream PCIe extension card. Like back in days we had separate graphics card, audio card etc. Now AI card. So to upgrade the PC to latest model, we could buy a new card, load up the drivers and boom, intelligence upgrade of the PC. This would be so cool.
This is exactly what's going to happen. Assuming no civilization-crippling or Great Filter events, anyway. At this point I fail to see how it could go any other way. The path has already been traveled, and governments (along with many other large organizations) will demand this functionality for themselves, which will eventually have a consumer market as well.
Another commenter mentioned how we keep cycling between local and server-based compute/storage as the dominant approach, and the cycle itself seems to be almost a law of nature. Nonetheless, regardless of where we're currently at in the cycle, there will always be both large and small players who want everything on-prem as much as possible.