At this point, numerous exotic mathematical tools from Langlands program, Lie algebras, algebraic geometry and topology have been used in physics.
I do agree that you can go quite far with calculus, linear algebra, and probability. But I do think that you overstate the case.
Algebraic geometry isn't any more complicated than multi-variable calculus. It's certainly more abstract but you get used to it after a while. Schemes¹ are not difficult to learn if you know basic commutative algebra & topology. If you don't know basic commutative algebra then you'll have a tougher time but it's again something that can be learned w/ enough practice & patience.
¹https://webhomes.maths.ed.ac.uk/~v1ranick/papers/eisenbudhar...