I'm not a physicist either but this stuff isn't magic. Most of the mathematics used by physicists isn't complicated if you've managed to get past calculus.
Algebraic geometry isn't any more complicated than multi-variable calculus. It's certainly more abstract but you get used to it after a while. Schemes¹ are not difficult to learn if you know basic commutative algebra & topology. If you don't know basic commutative algebra then you'll have a tougher time but it's again something that can be learned w/ enough practice & patience.
I'm not a physicist either but this stuff isn't magic. Most of the mathematics used by physicists isn't complicated if you've managed to get past calculus.
At this point, numerous exotic mathematical tools from Langlands program, Lie algebras, algebraic geometry and topology have been used in physics.
I do agree that you can go quite far with calculus, linear algebra, and probability. But I do think that you overstate the case.
Algebraic geometry isn't any more complicated than multi-variable calculus. It's certainly more abstract but you get used to it after a while. Schemes¹ are not difficult to learn if you know basic commutative algebra & topology. If you don't know basic commutative algebra then you'll have a tougher time but it's again something that can be learned w/ enough practice & patience.
¹https://webhomes.maths.ed.ac.uk/~v1ranick/papers/eisenbudhar...