OpenSSL 1.1.1b - TLS server handshake fails when using CAPI engine - capi_rsa_priv_enc() - capi engine: function not supported

Previous Topic Next Topic
 
classic Classic list List threaded Threaded
2 messages Options
Reply | Threaded
Open this post in threaded view
|

OpenSSL 1.1.1b - TLS server handshake fails when using CAPI engine - capi_rsa_priv_enc() - capi engine: function not supported

sandeep m.v
Hi,

This is regarding an issue reported here in this link - https://github.com/openssl/openssl/issues/8872  - This is blocking my development progress.
I'm seeing this issue - capi_rsa_priv_enc() - capi engine: function not supported, when I tried to upgrade my application from using OpenSSL version 1.0.2r to 1.1.1b.
This is causing TLS handshake to fail.
In my case, I'm creating a self signed certificate that is used by Server application loading CAPI engine.
Certificate is created by calling CertCreateSelfSignCertificate(--) - wincrypt.h - using szOID_RSA_SHA256RSA  signature algorithm and "Microsoft Enhanced RSA and AES Cryptographic Provider".

This failure doesn't look like it's because of TLS1.3 as turning off TLS1.3 while configure with "no-tls1_3" also caused the same problem.
Here is the call stack that is causing the reported problem when SSL_accept() is called.
 capi.dll!capi_rsa_priv_enc(int flen, const unsigned char * from, unsigned char * to, rsa_st * rsa, int padding)
libcrypto-1_1.dll!RSA_private_encrypt(int flen, const unsigned char * from, unsigned char * to, rsa_st * rsa, int padding)
libcrypto-1_1.dll!pkey_rsa_sign(evp_pkey_ctx_st * ctx, unsigned char * sig, unsigned int * siglen, const unsigned char * tbs, unsigned int tbslen)
libcrypto-1_1.dll!EVP_PKEY_sign(evp_pkey_ctx_st * ctx, unsigned char * sig, unsigned int * siglen, const unsigned char * tbs, unsigned int tbslen)
libcrypto-1_1.dll!EVP_DigestSignFinal(evp_md_ctx_st * ctx, unsigned char * sigret, unsigned int * siglen)
libcrypto-1_1.dll!EVP_DigestSign(evp_md_ctx_st * ctx, unsigned char * sigret, unsigned int * siglen, const unsigned char * tbs, unsigned int tbslen)
libssl-1_1.dll!tls_construct_cert_verify(ssl_st * s, wpacket_st * pkt)
libssl-1_1.dll!write_state_machine(ssl_st * s)
libssl-1_1.dll!state_machine(ssl_st * s, int server)
libssl-1_1.dll!ossl_statem_accept(ssl_st * s)
libssl-1_1.dll!SSL_do_handshake(ssl_st * s)
libssl-1_1.dll!SSL_accept(ssl_st * s)


Is there a solution for this? Or
Do I need to switch to some other Signature algorithm like ECDSA? Can I use this or anything else instead of RSA?
If I should switch to ECDSA, should I use "szOID_ECDSA_SHA256" (wincrypt.h) as signature algorithm, use "PROV_EC_ECDSA_SIG" while calling CryptAcquireContext() and call ENGINE_set_default() with ENGINE_METHOD_EC to support ECDSA using capi.dll?

Thank you in advance.

--
Regards,
Sandeep

Reply | Threaded
Open this post in threaded view
|

RE: OpenSSL 1.1.1b - TLS server handshake fails when using CAPI engine - capi_rsa_priv_enc() - capi engine: function not supported

Michael Wojcik
> From: openssl-users [mailto:[hidden email]] On Behalf Of sandeep m.v
> Sent: Monday, July 15, 2019 11:56

> I'm seeing this issue - capi_rsa_priv_enc() - capi engine: function not supported,
> when I tried to upgrade my application from using OpenSSL version 1.0.2r to 1.1.1b.
> This is causing TLS handshake to fail.
> In my case, I'm creating a self signed certificate that is used by Server application
> loading CAPI engine.
> Certificate is created by calling CertCreateSelfSignCertificate(--) - wincrypt.h -
> using szOID_RSA_SHA256RSA  signature algorithm and "Microsoft Enhanced RSA and
> AES Cryptographic Provider".

It's been some years since I worked with OpenSSL CAPI support, and in particular I haven't done anything with the CAPI engine for 1.1.

For 1.0.2, though, I ended up forking the OpenSSL CAPI engine support and enhancing it in a number of places. I changed capi_load_privkey and the functions it calls (capi_find_key and capi_get_pkey) to silently determine if the provider type in the context was wrong, and if so correct it.

CAPI is a fairly horrible API (CNG is somewhat better), and in particular is very fragile when there's a mismatch between the provider type in the CAPI context and the provider type specified for the key. It may be that your CAPI context specifies a provider other than the Enhanced RSA and AES one.

The CAPI engine for 1.0.2 (at least the version I forked from) also had a shortcoming which Steve Henson had suggested a fix for, but which wasn't in the code. That's down in capi_get_key where it calls CryptGetUserKey. If CryptGetUserKey fails with NTE_NO_KEY (you have to call GetLastError to check), then try again with the other key type, by XORing keyspec with 3:

   if (!CryptGetUserKey(key->hprov, keyspec, &key->key))
      {
      success = FALSE;
      errcode = GetLastError();
      if (errcode == NTE_NO_KEY)
        {
        keyspec ^= 3;
        success = CryptGetUserKey(key->hprov, keyspec, &key->key);
        }
      ...
      }

This handles the case where you have an RSA key marked as a key-exchange key and you need a signing key, or vice versa.

All of that may be fixed already in the 1.1 CAPI engine, or be irrelevant to your problem, of course. But those are potential issues when using RSA keys with CAPI.

--
Michael Wojcik
Distinguished Engineer, Micro Focus