The math stays difficult after basic discrete concepts and gets more difficult as you go. :)
It's straightforward to get yourself to a place where you can do cryptographic things and feel somewhat comfortable with what's happening. Truly understanding it to the point where you can reason safely about it is deceptively harder.
yeah I generally would say that learning about the actual schemes (tends to be) doable by a casual enthusiast, but learning about how the SOTA attacks work (which motivate scheme design for sure) is much more difficult.