I am playing around with RSA signatures with different padding and have some questions.
I have my sample code below for reference. It's in Swift (but it should still be close enough to C to be readable). Also in Swift, some of the complex macros in OpenSSL have to be broken down to be compilable hence my usage of EVP_DigestUpdate instead of EVP_DigestVerifyUpdate .
I am trying to define different padding options and so am defining and using a EVP_PKEY_CTX . However I am not sure if this padding is getting used in the signature since my Verify outputs OK regardless of which option my Sign uses. Which leads to:
1 - Do I need to use the same EVP_PKEY_CTX with the same options when doing verify? Right now even when I don't use any EVP_PKEY_CTX in Verify, I still verify OK.
2 - Do I need to set the hash function I am using in both EVP_PKEY_CTX as well as EVP_MD_CTX ? Or the latter is what defines this?
3 - In general, is there a way of making the Signature/Encryptions in OpenSSL be deterministic for debugging/testing purposes?