You're mistaking the thing for the tool we use to describe the thing.

Physics gives us a way to answer questions about nature, but it is not nature itself. It is also, so far (and probably forever), incomplete.

Math doesn't need to agree with nature, we can take it as far as we want, as long as it doesn't break its own rules. Physics uses it, but is not based on it.