Assertion in ssl_free_wbio_buffer() fails after unfinished handshake since OpenSSL 1.1.0
we encountered a strange behaviour of OpenSSL 1.1.0 when our test with "sslscan" provokes an unfinished
Our asynchronous communication approach uses memory BIOs in order to read/write data from other
communication layers into the SSL object. After the read/write operations are done the BIOs are freed.
Our example code is basically as follows:
err = SSL_accept (ssl);
BIO *rBIO = BIO_new(BIO_s_mem());
BIO *wBIO = BIO_new(BIO_s_mem());
SSL_set_bio(ssl, rBIO, wBIO);
// ... operate with read / write BIOs and SSL_read/SSL_write
SSL_set_bio(ssl, NULL, NULL); // free BIOs when finished
When calling SSL_free() after a complete handshake, everything is fine, but when calling SSL_free() after an
unfinished handshake, the assertion "assert(s->wbio != NULL);" in ssl_free_wbio_buffer() fails:
void ssl_free_wbio_buffer(SSL *s)
/* callers ensure s is never null */
if (s->bbio == NULL)
s->wbio = BIO_pop(s->wbio);
assert(s->wbio != NULL); /* <- this assertion fails! */
s->bbio = NULL;
With a complete handshake the new attribute “bbio” is freed by calling the method tls_finish_handshake(),
which itself calls ssl_free_wbio_buffer(). When the handshake is not finished successfully, the “bbio” is not freed, and
therefore when calling SSL_free() the assert in ssl_free_wbio_buffer() fails.
The problem exists since OpenSSL 1.1.0 - there were no problems with OpenSSL 1.0.2x.