The world just is, regardless of what we think about it. Physics is our best attempt so far to understand and predict it at a low level, but it will always be incomplete.
Maths (and especially compsci!) are constructions by and for humans.
Is it any wonder it is as you describe? It would be odd if it was any other way.
My point is precisely that I was often taught physics as if it was mathematics, where there is in fact a profound ontological difference between the two.
Also, physics (the discipline) is also a construction by and for humans.
To find tools applicable to reality. Not to construct reality.