Unfortunately we can’t observe fundamental things like “what are the rules of physics and time at the beginning of the universe?”. We look for clues and make large assumptions, but given that the universe experienced a 10^78 factor expansion during the Big Bang, assuming that actually happened, then why would it make sense to assume that the rules of the universe today are the rules for the very early beginning of the universe? A strand of DNA would become 10 light years. Given that relativity redefined our understanding of basic physics but only applies as we approach the speed of light, it would stand to reason that the rules of physics would be different from our current models based on today’s observations when the matter of the universe is packed much more tightly together.
Entirely reasonable assumptions! Our models match surprisingly well though... The CMB has a blackbody spectrum that aligns with predictions, we see galaxies more or less when and where we'd expect them, stellar populations look like what we'd expect for a universe made of hot hydrogen, and more! It's not quite perfect, but modern physics explains stuff really quite well even billions of years ago!
The inflationary epoch where it expanded by 10^78 in volume happened in the first 10^-32 seconds. The furthest galaxy we can see (fairly poorly) is 300M years after the Big Bang. It's likely if time or the rules were different, 300M years was enough for things to mostly die down to steady state. And as you say, they match more or less but those errors could easily hide remnants of when things were different. Of course, these are all numbers that assume the Big Bang theory is correct which is difficult to impossible to falsify since we can't possibly observe or test anything from that long ago. We'll have to wait to see if refinements to our model that clear up contradictions change what we think about the beginning of the universe and other boundary conditions.
Modern physics is guided by those observations, they can't be then used as an argument for its veracity.
Let's say I walk into a room and observe someone writing a tally mark on a chalk board once every second. I count 4x10^17 tally marks. I might assume that 4x10^17 seconds ago that same person entered the room and started tallying. I might even observe for the next 4x10^17 seconds they continue to tally. Heck I might even see a recorder going that when I play back at what I assume to be 1x speed, has chalk scratches at regular intervals for 4x10^17 seconds. I still don't have any actual evidence that they started those 8x10^17 seconds ago.