This wasn't easy.
The math in it isn't hard, and nothing is very complex, but I lost count of the times I thought "shit, we're doing this wrong" or "this would be insane to get right". Also, I found some stuff missing, for example any other public-key system than RSA, PGP, or the XEX/XTS modes of AES (which are used a lot in storage systems).
But all in all, this seems to be the best book on the topic out there at the moment.