Regression in 1.1.1 against 1.1.0 in SSL_CTX_new

classic Classic list List threaded Threaded
9 messages Options
Reply | Threaded
Open this post in threaded view
|

Regression in 1.1.1 against 1.1.0 in SSL_CTX_new

Harald Koch
Hello list,

I have a TLS server which is started on demand in a multithreaded (pthread) application. The TLS server is one thread which is being started and stopped. At first start, the TLS server initialized with SSL_CTX_new with TLS_server_method works as expected, after cleaning up, eliminating the thread and starting it again at a later time in the same process, SSL_CTX_new returns NULL. I’ve been digging deeper into the initialization code, and found out that in crypto/threads_pthread.c, function CRYPTO_THREAD_set_local the call to pthread_setspecific returns a value != 0 (in my case: 22). The error queue of openSSL stays empty. The same code works with openSSL 1.1.0 in all versions.
Some posts googled state that before usage, be sure to run OPENSSL_init_ssl (which I do, even if not required to my analysis since it’s already called in one of the called functions deeper in the library).
Am I missing something in a multithreaded environment?

Regards,
Harald
Reply | Threaded
Open this post in threaded view
|

Re: Regression in 1.1.1 against 1.1.0 in SSL_CTX_new

Matt Caswell-2


On 16/04/2020 14:42, Harald Koch wrote:
> Hello list,
>
> I have a TLS server which is started on demand in a multithreaded (pthread) application. The TLS server is one thread which is being started and stopped. At first start, the TLS server initialized with SSL_CTX_new with TLS_server_method works as expected, after cleaning up, eliminating the thread and starting it again at a later time in the same process, SSL_CTX_new returns NULL. I’ve been digging deeper into the initialization code, and found out that in crypto/threads_pthread.c, function

What does your clean up code look like? Are you taking specific steps to
cleanup OpenSSL and if so what are they?

Matt



 CRYPTO_THREAD_set_local the call to pthread_setspecific returns a value
!= 0 (in my case: 22). The error queue of openSSL stays empty. The same
code works with openSSL 1.1.0 in all versions.
> Some posts googled state that before usage, be sure to run OPENSSL_init_ssl (which I do, even if not required to my analysis since it’s already called in one of the called functions deeper in the library).
> Am I missing something in a multithreaded environment?
>
> Regards,
> Harald
>
Reply | Threaded
Open this post in threaded view
|

Re: Regression in 1.1.1 against 1.1.0 in SSL_CTX_new

Harald Koch
Hi Matt,

Am 16.04.2020 um 16:29 schrieb Matt Caswell <[hidden email]>:
On 16/04/2020 14:42, Harald Koch wrote:
Hello list,

I have a TLS server which is started on demand in a multithreaded (pthread) application. The TLS server is one thread which is being started and stopped. At first start, the TLS server initialized with SSL_CTX_new with TLS_server_method works as expected, after cleaning up, eliminating the thread and starting it again at a later time in the same process, SSL_CTX_new returns NULL. I’ve been digging deeper into the initialization code, and found out that in crypto/threads_pthread.c, function
What does your clean up code look like? Are you taking specific steps to
cleanup OpenSSL and if so what are they?

I’m checking if my actually used SSL and CTX are up, and if so, cleanup them before thread killing:

    if(ssl != NULL) { // assigned by SSL_new before
        SSL_free(ssl);
        ssl = NULL;
    }
    /* Free the SSL_CTX structure */
    if(ctx != NULL) { // assigned by SSL_CTX_new before
        SSL_CTX_free(ctx);
        ctx = NULL;
    }

No other openSSL specific cleanup functions are called. The functions documented in https://wiki.openssl.org/index.php/Library_Initialization#Cleanup are not called.


CRYPTO_THREAD_set_local the call to pthread_setspecific returns a value
!= 0 (in my case: 22). The error queue of openSSL stays empty. The same
code works with openSSL 1.1.0 in all versions.
Some posts googled state that before usage, be sure to run OPENSSL_init_ssl (which I do, even if not required to my analysis since it’s already called in one of the called functions deeper in the library).
Am I missing something in a multithreaded environment?

Reply | Threaded
Open this post in threaded view
|

Re: Regression in 1.1.1 against 1.1.0 in SSL_CTX_new

Tomas Mraz-2
In reply to this post by Harald Koch
On Thu, 2020-04-16 at 15:42 +0200, Harald Koch wrote:

> Hello list,
>
> I have a TLS server which is started on demand in a multithreaded
> (pthread) application. The TLS server is one thread which is being
> started and stopped. At first start, the TLS server initialized with
> SSL_CTX_new with TLS_server_method works as expected, after cleaning
> up, eliminating the thread and starting it again at a later time in
> the same process, SSL_CTX_new returns NULL. I’ve been digging deeper
> into the initialization code, and found out that in
> crypto/threads_pthread.c, function CRYPTO_THREAD_set_local the call
> to pthread_setspecific returns a value != 0 (in my case: 22). The
> error queue of openSSL stays empty. The same code works with openSSL
> 1.1.0 in all versions.
> Some posts googled state that before usage, be sure to run
> OPENSSL_init_ssl (which I do, even if not required to my analysis
> since it’s already called in one of the called functions deeper in
> the library).
> Am I missing something in a multithreaded environment?

Is this pure old 1.1.1 version or a current release from the 1.1.1
branch (i.e. 1.1.1f)?

Do you call the OPENSSL_init_ssl from the main thread or from the TLS
server thread?

--
Tomáš Mráz
No matter how far down the wrong road you've gone, turn back.
                                              Turkish proverb
[You'll know whether the road is wrong if you carefully listen to your
conscience.]


Reply | Threaded
Open this post in threaded view
|

Re: Regression in 1.1.1 against 1.1.0 in SSL_CTX_new

Harald Koch


> Am 16.04.2020 um 17:07 schrieb Tomas Mraz <[hidden email]>:
>
> On Thu, 2020-04-16 at 15:42 +0200, Harald Koch wrote:
>> Hello list,
>>
>> I have a TLS server which is started on demand in a multithreaded
>> (pthread) application. The TLS server is one thread which is being
>> started and stopped. At first start, the TLS server initialized with
>> SSL_CTX_new with TLS_server_method works as expected, after cleaning
>> up, eliminating the thread and starting it again at a later time in
>> the same process, SSL_CTX_new returns NULL. I’ve been digging deeper
>> into the initialization code, and found out that in
>> crypto/threads_pthread.c, function CRYPTO_THREAD_set_local the call
>> to pthread_setspecific returns a value != 0 (in my case: 22). The
>> error queue of openSSL stays empty. The same code works with openSSL
>> 1.1.0 in all versions.
>> Some posts googled state that before usage, be sure to run
>> OPENSSL_init_ssl (which I do, even if not required to my analysis
>> since it’s already called in one of the called functions deeper in
>> the library).
>> Am I missing something in a multithreaded environment?
>
> Is this pure old 1.1.1 version or a current release from the 1.1.1
> branch (i.e. 1.1.1f)?
It’s 1.1.1f, also tested 1.1.1c. In 1.1.0t it works. I can test against other versions if you want to. As a speciality, I compile openSSL with gzip support („./config enable-zlib ...“, for support of compressed SMIME contents in other application).

> Do you call the OPENSSL_init_ssl from the main thread or from the TLS
> server thread?

I call it from the TLS server thread (created by pthread_create):

if (!OPENSSL_init_ssl(OPENSSL_INIT_LOAD_SSL_STRINGS, NULL))
        return;
I tried to do it only once (instead of every started thread): no difference.

In addition, I load random data via /dev/urandom (also tested only once or every time the server thread starts):
        RAND_load_file("/dev/urandom", 256);


Reply | Threaded
Open this post in threaded view
|

Re: Regression in 1.1.1 against 1.1.0 in SSL_CTX_new

Tomas Mraz-2
On Thu, 2020-04-16 at 17:32 +0200, Harald Koch wrote:

> > Am 16.04.2020 um 17:07 schrieb Tomas Mraz <[hidden email]>:
> >
> > On Thu, 2020-04-16 at 15:42 +0200, Harald Koch wrote:
> > > Hello list,
> > >
> > > I have a TLS server which is started on demand in a multithreaded
> > > (pthread) application. The TLS server is one thread which is
> > > being
> > > started and stopped. At first start, the TLS server initialized
> > > with
> > > SSL_CTX_new with TLS_server_method works as expected, after
> > > cleaning
> > > up, eliminating the thread and starting it again at a later time
> > > in
> > > the same process, SSL_CTX_new returns NULL. I’ve been digging
> > > deeper
> > > into the initialization code, and found out that in
> > > crypto/threads_pthread.c, function CRYPTO_THREAD_set_local the
> > > call
> > > to pthread_setspecific returns a value != 0 (in my case: 22). The

This is EINVAL - meaning most probably that the pthread_setspecific()
is called on uninitialized key.

> > > error queue of openSSL stays empty. The same code works with
> > > openSSL
> > > 1.1.0 in all versions.
> > > Some posts googled state that before usage, be sure to run
> > > OPENSSL_init_ssl (which I do, even if not required to my analysis
> > > since it’s already called in one of the called functions deeper
> > > in
> > > the library).
> > > Am I missing something in a multithreaded environment?
> >
> > Is this pure old 1.1.1 version or a current release from the 1.1.1
> > branch (i.e. 1.1.1f)?
> It’s 1.1.1f, also tested 1.1.1c. In 1.1.0t it works. I can test
> against other versions if you want to. As a speciality, I compile
> openSSL with gzip support („./config enable-zlib ...“, for support of
> compressed SMIME contents in other application).
>
> > Do you call the OPENSSL_init_ssl from the main thread or from the
> > TLS
> > server thread?
>
> I call it from the TLS server thread (created by pthread_create):
>
> if (!OPENSSL_init_ssl(OPENSSL_INIT_LOAD_SSL_STRINGS, NULL))
> return;
> I tried to do it only once (instead of every started thread): no
> difference.

I do not see how this error could really happen unless
OPENSSL_cleanup() is called.

Could you try to set a breakpoint on that function and see if it is
somehow called inadvertently?

> In addition, I load random data via /dev/urandom (also tested only
> once or every time the server thread starts):
> RAND_load_file("/dev/urandom", 256);

That call should not be necessary.

--
Tomáš Mráz
No matter how far down the wrong road you've gone, turn back.
                                              Turkish proverb
[You'll know whether the road is wrong if you carefully listen to your
conscience.]


Reply | Threaded
Open this post in threaded view
|

Re: Regression in 1.1.1 against 1.1.0 in SSL_CTX_new

Harald Koch
Am 16.04.2020 um 17:54 schrieb Tomas Mraz <[hidden email]>:

error queue of openSSL stays empty. The same code works with
openSSL with gzip support („./config enable-zlib ...“, for support of
compressed SMIME contents in other application).
Do you call the OPENSSL_init_ssl from the main thread or from the
TLS
server thread?
I call it from the TLS server thread (created by pthread_create):

if (!OPENSSL_init_ssl(OPENSSL_INIT_LOAD_SSL_STRINGS, NULL))
return;
I tried to do it only once (instead of every started thread): no
difference.
I do not see how this error could really happen unless
OPENSSL_cleanup() is called.

Could you try to set a breakpoint on that function and see if it is
somehow called inadvertently?
gdb is actually unavailable, so I added a big „printf“ at the beginning of crypto/init.c, recompiled (even without zlib support as I’ve seen there’s much functionality in there), function OPENSSL_cleanup: it’s not called. I’m very sure OPENSSL_cleanup is not called. 

In addition, I load random data via /dev/urandom (also tested only
once or every time the server thread starts):
RAND_load_file("/dev/urandom", 256);
That call should not be necessary.
I removed it just in case it may have an influence. No better result. :(


I have a workaround and possibly it’s the solution, I would like to have your valuable statement here: you asked where I call OPENSSL_init_ssl: it’s done in the thread for TLS server. When I initialize it earlier in the main thread, the subsequent generated (second) thread works as expected! Is this spooky or expected behaviour?
Reply | Threaded
Open this post in threaded view
|

Re: Regression in 1.1.1 against 1.1.0 in SSL_CTX_new

OpenSSL - User mailing list
On Thu, Apr 16, 2020 at 09:41:23PM +0200, Harald Koch wrote:

> Am 16.04.2020 um 17:54 schrieb Tomas Mraz <[hidden email]>:
> >
> > error queue of openSSL stays empty. The same code works with
> >> openSSL with gzip support („./config enable-zlib ...“, for support of
> >> compressed SMIME contents in other application).
> >> Do you call the OPENSSL_init_ssl from the main thread or from the
> >>> TLS
> >>> server thread?
> >> I call it from the TLS server thread (created by pthread_create):
> >>
> >> if (!OPENSSL_init_ssl(OPENSSL_INIT_LOAD_SSL_STRINGS, NULL))
> >> return;
> >> I tried to do it only once (instead of every started thread): no
> >> difference.
> > I do not see how this error could really happen unless
> > OPENSSL_cleanup() is called.
> >
> > Could you try to set a breakpoint on that function and see if it is
> > somehow called inadvertently?
> gdb is actually unavailable, so I added a big „printf“ at the beginning of crypto/init.c, recompiled (even without zlib support as I’ve seen there’s much functionality in there), function OPENSSL_cleanup: it’s not called. I’m very sure OPENSSL_cleanup is not called.
>
> > In addition, I load random data via /dev/urandom (also tested only
> >> once or every time the server thread starts):
> >> RAND_load_file("/dev/urandom", 256);
> > That call should not be necessary.
> I removed it just in case it may have an influence. No better result. :(
>
>
> I have a workaround and possibly it’s the solution, I would like to have your valuable statement here: you asked where I call OPENSSL_init_ssl: it’s done in the thread for TLS server. When I initialize it earlier in the main thread, the subsequent generated (second) thread works as expected! Is this spooky or expected behaviour?

Just to check: what OS is this on?

-Ben
Reply | Threaded
Open this post in threaded view
|

Re: Regression in 1.1.1 against 1.1.0 in SSL_CTX_new

Harald Koch


> Am 16.04.2020 um 22:17 schrieb Benjamin Kaduk <[hidden email]>:
>
> On Thu, Apr 16, 2020 at 09:41:23PM +0200, Harald Koch wrote:
>> Am 16.04.2020 um 17:54 schrieb Tomas Mraz <[hidden email]>:
>>>
>>> error queue of openSSL stays empty. The same code works with
>>>> openSSL with gzip support („./config enable-zlib ...“, for support of
>>>> compressed SMIME contents in other application).
>>>> Do you call the OPENSSL_init_ssl from the main thread or from the
>>>>> TLS
>>>>> server thread?
>>>> I call it from the TLS server thread (created by pthread_create):
>>>>
>>>> if (!OPENSSL_init_ssl(OPENSSL_INIT_LOAD_SSL_STRINGS, NULL))
>>>> return;
>>>> I tried to do it only once (instead of every started thread): no
>>>> difference.
>>> I do not see how this error could really happen unless
>>> OPENSSL_cleanup() is called.
>>>
>>> Could you try to set a breakpoint on that function and see if it is
>>> somehow called inadvertently?
>> gdb is actually unavailable, so I added a big „printf“ at the beginning of crypto/init.c, recompiled (even without zlib support as I’ve seen there’s much functionality in there), function OPENSSL_cleanup: it’s not called. I’m very sure OPENSSL_cleanup is not called.
>>
>>> In addition, I load random data via /dev/urandom (also tested only
>>>> once or every time the server thread starts):
>>>> RAND_load_file("/dev/urandom", 256);
>>> That call should not be necessary.
>> I removed it just in case it may have an influence. No better result. :(
>>
>>
>> I have a workaround and possibly it’s the solution, I would like to have your valuable statement here: you asked where I call OPENSSL_init_ssl: it’s done in the thread for TLS server. When I initialize it earlier in the main thread, the subsequent generated (second) thread works as expected! Is this spooky or expected behaviour?
>
> Just to check: what OS is this on?
Linux x86-64. Every Debian from 7 to 10 tested.