The attack being physical doesn't mean you need physical access.
You can affect memory cells for example by repeatedly writing to cells next to them.
So if you have write access to some part of memory, you can affect memory you might not have access to because of hardware effects. This breaks assumptions in software but can be done completely remotely.
Thanks, that's an important point.
I guess that branch prediction attacks are an analogous phenomenon, although slightly less physical.