This. I went into my 400 level cryptography class with 0 Number Theory knowledge as it wasn't a prerequisite. Big mistake, it should've been. My lowest grade in college and I still busted my ass.
That being said, it was incredibly interesting and I do not regret it.
I had Boldyreva back in 2012-ish. She's an excellent instructor who knows the material well. The course material is quite challenging though. You'll have to be able to pick out attack methods or prove security on custom cryptosystems. Be prepared to put in many many hours of studying if this is an area which is unfamiliar to you.