in general the math is not actually that hard. It will be things you don't know beforehand, but a general undergraduate cryptography class will not assume the undergraduates have that much of a better math background than you. Typically just
1. comfort with logical operations/arithmetic over F2 2. discrete probability over finite sets 3. some basic complexity theory (mostly to reason about running time, though being familiar with proofs by reduction can help as well if you actually want to do security proofs).
a decent idea might be to take some "good" undergraduate cryptography class's course resource and use that. For example, Mihir Bellare is an extremely accomplished cryptographer. The course materials for his undergrad course F2018 are
https://cseweb.ucsd.edu/~mihir/cse107/slides.html
He's also written a longer series of lecture notes on cryptography that's freely available. I don't know where it is on his webpage these days, but you can find it below
https://www.cs.tufts.edu/comp/165/papers/Goldwasser-Bellare-...
the difficult part with this approach is not being able to ask questions that easily. To "fix" this, you can either
* use AI, though that has its own issues, or * use some community forum, such as crypto.stackexchange.com
if you want a full book, the typical (undergradute) one that roughly matches the above syllabus is "An Introduction to Modern Cryptograph" by Katz and Lindell.
I've also heard good things about Mike Roseluk's the joy of cryptography
https://joyofcryptography.com/
Boneh and Shoup have a decent (freely available, and very comprehensive) textbook at the graduate level
but it is following (roughly) the standard undergraduate curriculum, so if the slides I linked too are too sparse at some point, you could look up that topic in Boneh and Shoup (or use Boneh and Shoup as context to ask an LLM more targeted questions).
That all being said, the main difficulty for someone in your position is likely determining "what to learn" in cryptography. The easy thing would be to follow the standard undergraduate track, but if you're interested in any particular topic there are likely better routes to take.