X509_verify_cert() and multiple threads

classic Classic list List threaded Threaded
13 messages Options
Reply | Threaded
Open this post in threaded view
|

X509_verify_cert() and multiple threads

Steffen Fiksdal
Hi!

I have a library using openssl 0.9.8 that is accessed by several threads.

One function verifies a certificate, and in that function I call
X509_verify_cert().

In a few occations the function returns 0, and the failure occures in the
file "rsa_pk1.c" on line number 100.
I tried to do a mutex around the X509_verify_cert call and never
experienced the error.
Now I believe I have set up threading properly with CRYPTO_num_locks()
number of mutexes with a proper locking callback with
CRYPTO_set_locking_callback().

I use *a* lot of openssl in my library and have tested it extensively for
threading issues.
This problem is the only threading issue I can se at the moment.

Any ideas what I do wrong?

Best regards
Steffen Fiksdal
______________________________________________________________________
OpenSSL Project                                 http://www.openssl.org
User Support Mailing List                    [hidden email]
Automated List Manager                           [hidden email]
Reply | Threaded
Open this post in threaded view
|

Re: X509_verify_cert() and multiple threads

Dr. Stephen Henson
On Wed, Nov 09, 2005, Steffen Fiksdal wrote:

> Hi!
>
> I have a library using openssl 0.9.8 that is accessed by several threads.
>
> One function verifies a certificate, and in that function I call
> X509_verify_cert().
>
> In a few occations the function returns 0, and the failure occures in the
> file "rsa_pk1.c" on line number 100.
> I tried to do a mutex around the X509_verify_cert call and never
> experienced the error.
> Now I believe I have set up threading properly with CRYPTO_num_locks()
> number of mutexes with a proper locking callback with
> CRYPTO_set_locking_callback().
>
> I use *a* lot of openssl in my library and have tested it extensively for
> threading issues.
> This problem is the only threading issue I can se at the moment.
>
> Any ideas what I do wrong?
>

It is possible the cause is the X509_PUBKEY_get() function which I've noticed
has a possible race condition when several threads initially try to get the
certificate public key and try to decode it.

Can you try placing a mutex round the X509_get_pubkey() call in x509_cmp.c to
see if that is the cause?

Steve.
--
Dr Stephen N. Henson. Email, S/MIME and PGP keys: see homepage
OpenSSL project core developer and freelance consultant.
Funding needed! Details on homepage.
Homepage: http://www.drh-consultancy.demon.co.uk
______________________________________________________________________
OpenSSL Project                                 http://www.openssl.org
User Support Mailing List                    [hidden email]
Automated List Manager                           [hidden email]
Reply | Threaded
Open this post in threaded view
|

Re: X509_verify_cert() and multiple threads

Steffen Fiksdal

> On Wed, Nov 09, 2005, Steffen Fiksdal wrote:
>
>> Hi!
>>
>> I have a library using openssl 0.9.8 that is accessed by several threads.
>>
>> One function verifies a certificate, and in that function I call
>> X509_verify_cert().
>>
>> In a few occations the function returns 0, and the failure occures in the
>> file "rsa_pk1.c" on line number 100.
>> I tried to do a mutex around the X509_verify_cert call and never
>> experienced the error.
>> Now I believe I have set up threading properly with CRYPTO_num_locks()
>> number of mutexes with a proper locking callback with
>> CRYPTO_set_locking_callback().
>>
>> I use *a* lot of openssl in my library and have tested it extensively for
>> threading issues.
>> This problem is the only threading issue I can se at the moment.
>>
>> Any ideas what I do wrong?
>>
>
> It is possible the cause is the X509_PUBKEY_get() function which I've noticed
> has a possible race condition when several threads initially try to get the
> certificate public key and try to decode it.
>
> Can you try placing a mutex round the X509_get_pubkey() call in x509_cmp.c to
> see if that is the cause?
>

I mutexed the call with CRYPTO_w_lock/CRYPTO_w_unlock, but the error still
pops up from time to time.

I am investigating, and will come back when I find out what happens.

Best regards
Steffen Fiksdal
______________________________________________________________________
OpenSSL Project                                 http://www.openssl.org
User Support Mailing List                    [hidden email]
Automated List Manager                           [hidden email]
Reply | Threaded
Open this post in threaded view
|

Re: X509_verify_cert() and multiple threads

Dr. Stephen Henson
On Wed, Nov 09, 2005, Steffen Fiksdal wrote:

>
>
> I am investigating, and will come back when I find out what happens.
>

OK, I'd suggest placing a couple of calls in the rsa_pk1.c file which dumps
out the public key and the block it is attempting to decrypt.

Then see if the public/key and/or the block values are consistent with those
in the certificates.

Steve.
--
Dr Stephen N. Henson. Email, S/MIME and PGP keys: see homepage
OpenSSL project core developer and freelance consultant.
Funding needed! Details on homepage.
Homepage: http://www.drh-consultancy.demon.co.uk
______________________________________________________________________
OpenSSL Project                                 http://www.openssl.org
User Support Mailing List                    [hidden email]
Automated List Manager                           [hidden email]
Reply | Threaded
Open this post in threaded view
|

Re: X509_verify_cert() and multiple threads

Steffen Fiksdal

> On Wed, Nov 09, 2005, Steffen Fiksdal wrote:
>
>>
>>
>> I am investigating, and will come back when I find out what happens.
>>
>
> OK, I'd suggest placing a couple of calls in the rsa_pk1.c file which dumps
> out the public key and the block it is attempting to decrypt.
>
> Then see if the public/key and/or the block values are consistent with those
> in the certificates.
>


Hi!

I have investigated some more on this issue.
I have traced the problem down to the MONT_HELPER call int the
RSA_eay_public_decrypt function in file rsa_eay.c.
If I mutex this call everything works fine, if I don't the verifications
failes in a few occations.

So it seems that the MONT_HELPER has a threading issue ?

Best Regards
Steffen Fiksdal
______________________________________________________________________
OpenSSL Project                                 http://www.openssl.org
User Support Mailing List                    [hidden email]
Automated List Manager                           [hidden email]
Reply | Threaded
Open this post in threaded view
|

Re: X509_verify_cert() and multiple threads

Dr. Stephen Henson
On Thu, Nov 10, 2005, Steffen Fiksdal wrote:

>
> I have investigated some more on this issue.
> I have traced the problem down to the MONT_HELPER call int the
> RSA_eay_public_decrypt function in file rsa_eay.c.
> If I mutex this call everything works fine, if I don't the verifications
> failes in a few occations.
>
> So it seems that the MONT_HELPER has a threading issue ?
>

What OS are you using and is it multi processor or does it use hyperthreading?


Steve.
--
Dr Stephen N. Henson. Email, S/MIME and PGP keys: see homepage
OpenSSL project core developer and freelance consultant.
Funding needed! Details on homepage.
Homepage: http://www.drh-consultancy.demon.co.uk
______________________________________________________________________
OpenSSL Project                                 http://www.openssl.org
User Support Mailing List                    [hidden email]
Automated List Manager                           [hidden email]
Reply | Threaded
Open this post in threaded view
|

Re: X509_verify_cert() and multiple threads

Steffen Fiksdal

> On Thu, Nov 10, 2005, Steffen Fiksdal wrote:
>
>>
>> I have investigated some more on this issue.
>> I have traced the problem down to the MONT_HELPER call int the
>> RSA_eay_public_decrypt function in file rsa_eay.c.
>> If I mutex this call everything works fine, if I don't the verifications
>> failes in a few occations.
>>
>> So it seems that the MONT_HELPER has a threading issue ?
>>
>
> What OS are you using and is it multi processor or does it use hyperthreading?


It seems strange that this small macro should create any threading issues.

I use Linux 2.4.21-27 SMP. It has two Intel  processors, but only one is
physically in use. No hyperthreading.

I should try this on a non-SMP kernel and check ?
Maybe it's the pthread implementation that does not cope well with SMP.

Best Regards
Steffen

______________________________________________________________________
OpenSSL Project                                 http://www.openssl.org
User Support Mailing List                    [hidden email]
Automated List Manager                           [hidden email]
Reply | Threaded
Open this post in threaded view
|

Re: X509_verify_cert() and multiple threads

Dr. Stephen Henson
On Thu, Nov 10, 2005, Steffen Fiksdal wrote:

>
> >On Thu, Nov 10, 2005, Steffen Fiksdal wrote:
> >
> >>
> >>I have investigated some more on this issue.
> >>I have traced the problem down to the MONT_HELPER call int the
> >>RSA_eay_public_decrypt function in file rsa_eay.c.
> >>If I mutex this call everything works fine, if I don't the verifications
> >>failes in a few occations.
> >>
> >>So it seems that the MONT_HELPER has a threading issue ?
> >>
> >
> >What OS are you using and is it multi processor or does it use
> >hyperthreading?
>
>
> It seems strange that this small macro should create any threading issues.
>
> I use Linux 2.4.21-27 SMP. It has two Intel  processors, but only one is
> physically in use. No hyperthreading.
>
> I should try this on a non-SMP kernel and check ?
> Maybe it's the pthread implementation that does not cope well with SMP.
>

My reason for asking is that there has been some discussion suggesting that
OpenSSLs use of DCLP (double-checked locking pattern) may not be valid for
some environments. IIRC multi processors and hyperthreading were two cases
which might cause problems.

No one has so far come up with a concrete example of it failing though.

Steve.
--
Dr Stephen N. Henson. Email, S/MIME and PGP keys: see homepage
OpenSSL project core developer and freelance consultant.
Funding needed! Details on homepage.
Homepage: http://www.drh-consultancy.demon.co.uk
______________________________________________________________________
OpenSSL Project                                 http://www.openssl.org
User Support Mailing List                    [hidden email]
Automated List Manager                           [hidden email]
Reply | Threaded
Open this post in threaded view
|

Re: X509_verify_cert() and multiple threads

Steffen Fiksdal

> On Thu, Nov 10, 2005, Steffen Fiksdal wrote:
>
>>
>>> On Thu, Nov 10, 2005, Steffen Fiksdal wrote:
>>>
>>>>
>>>> I have investigated some more on this issue.
>>>> I have traced the problem down to the MONT_HELPER call int the
>>>> RSA_eay_public_decrypt function in file rsa_eay.c.
>>>> If I mutex this call everything works fine, if I don't the verifications
>>>> failes in a few occations.
>>>>
>>>> So it seems that the MONT_HELPER has a threading issue ?
>>>>
>>>
>>> What OS are you using and is it multi processor or does it use
>>> hyperthreading?
>>
>>
>> It seems strange that this small macro should create any threading issues.
>>
>> I use Linux 2.4.21-27 SMP. It has two Intel  processors, but only one is
>> physically in use. No hyperthreading.
>>
>> I should try this on a non-SMP kernel and check ?
>> Maybe it's the pthread implementation that does not cope well with SMP.
>>
>
> My reason for asking is that there has been some discussion suggesting that
> OpenSSLs use of DCLP (double-checked locking pattern) may not be valid for
> some environments. IIRC multi processors and hyperthreading were two cases
> which might cause problems.
>
> No one has so far come up with a concrete example of it failing though.


I have tried to get this error on my other linux box, a uniprocessor non
SMP Linux kernel, without luck.

So I experience the error in a few occations on the SMP kernel, but not on
the uniprocessor non SMP Linux kernel.

When I mutex the MONT_HELPER call myself by wrapping the macro with a
CRYPTPO_lock(), I do not get the error on the SMP kernel.

This can have several reasons, but it is kind of strange..

Best Regards,
Steffen Fiksdal
______________________________________________________________________
OpenSSL Project                                 http://www.openssl.org
User Support Mailing List                    [hidden email]
Automated List Manager                           [hidden email]
Reply | Threaded
Open this post in threaded view
|

Re: X509_verify_cert() and multiple threads

Dr. Stephen Henson
On Thu, Nov 10, 2005, Steffen Fiksdal wrote:

>
>
> I have tried to get this error on my other linux box, a uniprocessor non
> SMP Linux kernel, without luck.
>
> So I experience the error in a few occations on the SMP kernel, but not on
> the uniprocessor non SMP Linux kernel.
>
> When I mutex the MONT_HELPER call myself by wrapping the macro with a
> CRYPTPO_lock(), I do not get the error on the SMP kernel.
>
> This can have several reasons, but it is kind of strange..
>

On 0.9.8 that macro calls BN_MONT_CTX_set_locked(). Can you put some locks in
there so a lock is always called and see if that helps?

Steve.
--
Dr Stephen N. Henson. Email, S/MIME and PGP keys: see homepage
OpenSSL project core developer and freelance consultant.
Funding needed! Details on homepage.
Homepage: http://www.drh-consultancy.demon.co.uk
______________________________________________________________________
OpenSSL Project                                 http://www.openssl.org
User Support Mailing List                    [hidden email]
Automated List Manager                           [hidden email]
Reply | Threaded
Open this post in threaded view
|

Re: X509_verify_cert() and multiple threads

Steffen Fiksdal

> On Thu, Nov 10, 2005, Steffen Fiksdal wrote:
>
>>
>>
>> I have tried to get this error on my other linux box, a uniprocessor non
>> SMP Linux kernel, without luck.
>>
>> So I experience the error in a few occations on the SMP kernel, but not on
>> the uniprocessor non SMP Linux kernel.
>>
>> When I mutex the MONT_HELPER call myself by wrapping the macro with a
>> CRYPTPO_lock(), I do not get the error on the SMP kernel.
>>
>> This can have several reasons, but it is kind of strange..
>>
>
> On 0.9.8 that macro calls BN_MONT_CTX_set_locked(). Can you put some locks in
> there so a lock is always called and see if that helps?


I serialized the whole BN_MONT_CTX_set_locked() function, but it still
fails.
THe BN_MONT_CTX_set_locked is not always called because sometimes
the _method_mod_n is NOT NULL.

Because the BN_MONT_CTX_set_locked seems to work, this race condition must
be for the times the BN_MONT_CTX_set_locked() is not called.

I do not have knowledge about the details here, but I report what I
experience..

I have the sequence for when it fails. First I print "Going MONT_HELPER"
just before the macro is called, then I print "Went MONT_HELPER" just
after the macro is finished. I also print the pre_cond and _method_mod_n
values. The result was the following with 2 threads:


Going MONT_HELPER
pre_cond=2
_method_mod_n == NULL
Going MONT_HELPER
pre_cond=2
_method_mod_n != NULL
Went MONT_HELPER
Went MONT_HELPER.

The times it works ok, one thread goes through MONT_HELPER without any
disturbance from other threads.  In this error situation this was not the
case. For the second thread in the output the BN_MONT_CTX_set_locked()
will not be called.

Best Regards
Steffen Fiksdal


______________________________________________________________________
OpenSSL Project                                 http://www.openssl.org
User Support Mailing List                    [hidden email]
Automated List Manager                           [hidden email]
Reply | Threaded
Open this post in threaded view
|

Re: X509_verify_cert() and multiple threads

Dr. Stephen Henson
On Fri, Nov 11, 2005, Steffen Fiksdal wrote:

>
>
>
> I serialized the whole BN_MONT_CTX_set_locked() function, but it still
> fails.
> THe BN_MONT_CTX_set_locked is not always called because sometimes
> the _method_mod_n is NOT NULL.
>

OK, I've committed a fix. Please apply this patch:

http://cvs.openssl.org/chngview?cn=14619

or try the next snapshot.

Steve.
--
Dr Stephen N. Henson. Email, S/MIME and PGP keys: see homepage
OpenSSL project core developer and freelance consultant.
Funding needed! Details on homepage.
Homepage: http://www.drh-consultancy.demon.co.uk
______________________________________________________________________
OpenSSL Project                                 http://www.openssl.org
User Support Mailing List                    [hidden email]
Automated List Manager                           [hidden email]
Reply | Threaded
Open this post in threaded view
|

Re: X509_verify_cert() and multiple threads

Steffen Fiksdal


> On Fri, Nov 11, 2005, Steffen Fiksdal wrote:
>
>>
>>
>>
>> I serialized the whole BN_MONT_CTX_set_locked() function, but it still
>> fails.
>> THe BN_MONT_CTX_set_locked is not always called because sometimes
>> the _method_mod_n is NOT NULL.
>>
>
> OK, I've committed a fix. Please apply this patch:
>
> http://cvs.openssl.org/chngview?cn=14619
>
> or try the next snapshot.


Works like a dream!

Best Regards
Steffen Fiksdal
______________________________________________________________________
OpenSSL Project                                 http://www.openssl.org
User Support Mailing List                    [hidden email]
Automated List Manager                           [hidden email]