We’re using OpenSSl 1.1.1b on WIN64 and are facing a (rare but strange) core-dump when doing the following:
After successful TLS1.3 handshake we’re calling SSL_read() to get the first 2 Bytes of PDU data from the new connection (ASN.1 TAG + length).
SSL_read() returns 0
According to OpenSSL 1.1.1 documentation:
For SSL_read() and SSL_peek() the following return values can occur:
The read operation was not successful, because either the connection was closed, an error occurred or action must be taken
by the calling process. Call SSL_get_error(3) with the return value ret to find out the reason.
We follow and call SSL_get_error() which crashes with debugger output (only topmost frame is shown here)
LIBSSL!SSL_get_error(struct ssl_st * s = 0x00000000`05be9a00, int i = <Value unavailable error>)+0x18c [d:\data\openssl\64\openssl-1.1.1b\ssl\ssl_lib.c @ 3560]
The OpenSSL source at this reported line looks like:
if ((s->shutdown & SSL_RECEIVED_SHUTDOWN) &&
(s->s3->warn_alert == SSL_AD_CLOSE_NOTIFY))
when we inspect the session “s” in the debugger, we find that s->shutdown == 3 and s->s3 == NULL which finally causes the crash
It looks like a bug in OpenSSL….???
So far it happened only once in our LAB (after some hours of heavy SSL load testing with thousands of SSL connections created/deleted….)
we’re currently not able to reproduce it…..