It's not even just that. If you just built the rom chips separately and swapped them for the RAM of a normal accelerator, it would not help at all.

The trick is that every compute element in their system has it's own small pool of ROM, instead of putting all the ram behind a common pipe. ROM is just used because it's the densest kind of memory that can be fabricated on the same process as their logic.