libssl-1_1.dll!state_machine(ssl_st * s, int server)
libssl-1_1.dll!ossl_statem_accept(ssl_st * s)
libssl-1_1.dll!SSL_do_handshake(ssl_st * s)
libssl-1_1.dll!SSL_accept(ssl_st * s)
Is there a solution for this? Or
Do I need to switch to some other Signature algorithm like ECDSA? Can I use this or anything else instead of RSA?
If I should switch to ECDSA, should I use "szOID_ECDSA_SHA256" (wincrypt.h) as signature algorithm, use "PROV_EC_ECDSA_SIG" while calling CryptAcquireContext() andcall ENGINE_set_default() with ENGINE_METHOD_EC to support ECDSA using capi.dll?
RE: OpenSSL 1.1.1b - TLS server handshake fails when using CAPI engine - capi_rsa_priv_enc() - capi engine: function not supported
> From: openssl-users [mailto:[hidden email]] On Behalf Of sandeep m.v
> Sent: Monday, July 15, 2019 11:56
> I'm seeing this issue - capi_rsa_priv_enc() - capi engine: function not supported,
> when I tried to upgrade my application from using OpenSSL version 1.0.2r to 1.1.1b.
> This is causing TLS handshake to fail.
> In my case, I'm creating a self signed certificate that is used by Server application
> loading CAPI engine.
> Certificate is created by calling CertCreateSelfSignCertificate(--) - wincrypt.h -
> using szOID_RSA_SHA256RSA signature algorithm and "Microsoft Enhanced RSA and
> AES Cryptographic Provider".
It's been some years since I worked with OpenSSL CAPI support, and in particular I haven't done anything with the CAPI engine for 1.1.
For 1.0.2, though, I ended up forking the OpenSSL CAPI engine support and enhancing it in a number of places. I changed capi_load_privkey and the functions it calls (capi_find_key and capi_get_pkey) to silently determine if the provider type in the context was wrong, and if so correct it.
CAPI is a fairly horrible API (CNG is somewhat better), and in particular is very fragile when there's a mismatch between the provider type in the CAPI context and the provider type specified for the key. It may be that your CAPI context specifies a provider other than the Enhanced RSA and AES one.
The CAPI engine for 1.0.2 (at least the version I forked from) also had a shortcoming which Steve Henson had suggested a fix for, but which wasn't in the code. That's down in capi_get_key where it calls CryptGetUserKey. If CryptGetUserKey fails with NTE_NO_KEY (you have to call GetLastError to check), then try again with the other key type, by XORing keyspec with 3: