Modern physics is guided by those observations, they can't be then used as an argument for its veracity.

Let's say I walk into a room and observe someone writing a tally mark on a chalk board once every second. I count 4x10^17 tally marks. I might assume that 4x10^17 seconds ago that same person entered the room and started tallying. I might even observe for the next 4x10^17 seconds they continue to tally. Heck I might even see a recorder going that when I play back at what I assume to be 1x speed, has chalk scratches at regular intervals for 4x10^17 seconds. I still don't have any actual evidence that they started those 8x10^17 seconds ago.