I am playing around with RSA signatures with different padding options and I have some questions.
I am trying to define different padding options and so am defining and using a EVP_PKEY_CTX . However I am not sure if this padding is getting used in the signature since my Verify outputs OK regardless of which option my Sign uses. Which leads to:
1 - Do I need to use a EVP_PKEY_CTX with the same options when doing verify? If so, I assume I can't reuse the same PKey_Ctx and I have to define another one. Right now even when I don't use any EVP_PKEY_CTX in Verify, I still verify OK, which makes me question if the padding option has been set.
2 - Is there a way to figure out what padding/hashing/etc option was used for the Sign/verify operation? This way I can be sure what algorithm or standard is being used.
3 - Do I need to set the hash function I am using in both EVP_PKEY_CTX as well as EVP_MD_CTX ? Or the latter is what defines this for the signing option?
4 - In general, is there a way of making the Signature/Encryptions in OpenSSL be deterministic for debugging/testing purposes?
5 - I noticed that there are two ways of determining the signature size: (a) by calling EVP_PKEY_size(rsaKeypair) as I am doing below, as well as (b) calling EVP_DigestSignFinal(md_ctx, nil, &sig_len) . Is one better than the other?
My sample code is below for reference. It's in Swift (but it should still be close enough to C to be readable). Also in Swift, some of the complex macros in OpenSSL have to be broken down to be compilable hence my usage of EVP_DigestUpdate instead of EVP_DigestVerifyUpdate .
Thanks in advance for any insight in the above.
let md_ctx = EVP_MD_CTX_create()
let md_ctx_verify = EVP_MD_CTX_create()
// To define padding option used in signature
let pkey_ctx = EVP_PKEY_CTX_new(rsaKeypair, nil)
// EVP_PKEY_CTX_set_rsa_padding(pkey_ctx, RSA_PKCS1_PADDING) - complex macro needs to be replaced