Random and rare Seg faults at openssl library level

Previous Topic Next Topic
 
classic Classic list List threaded Threaded
18 messages Options
Reply | Threaded
Open this post in threaded view
|

Random and rare Seg faults at openssl library level

Gimhani Uthpala
Dear team,
I'm running an application which uses openssl for secure communication between processes. I am getting seg-faults at openssl level. This only occurred very randomly and the following are stacks that seg faults  at openssl level in the given 2 cases. We are using openssl 1.0.2k. 

Went through the security vulnerabilities list for this version but couldn't find a clue. Running valgrind too didn't give an exact clue related to the issue. Can you please guide me how can I find the exact root cause for the seg fault?  

I am calling SSL_do_handshake(ssl_ctx) from my code level and both the below seg faults are occuring from it's inside. 

#0  0x00007fd64cdabdd3 in ASN1_item_verify () from /lib64/libcrypto.so.10
#1  0x00007fd64cdcac58 in internal_verify () from /lib64/libcrypto.so.10
#2  0x00007fd64cdccaef in X509_verify_cert () from /lib64/libcrypto.so.10
#3  0x00007fd64d111c68 in ssl_verify_cert_chain () from /lib64/libssl.so.10
#4  0x00007fd64d0e8cc6 in ssl3_get_client_certificate () from /lib64/libssl.so.10
#5  0x00007fd64d0ea3f8 in ssl3_accept () from /lib64/libssl.so.10


 #0 0x00007ffb65529854 in RSA_verify () from /usr/lib64/libcrypto.so.10
 #1 0x00007ffb6552f1fc in pkey_rsa_verify () from /usr/lib64/libcrypto.so.10
 #2 0x00007ffb65561877 in EVP_DigestVerifyFinal () from /usr/lib64/libcrypto.so.10
 #3 0x00007ffb6556af25 in ASN1_item_verify () from /usr/lib64/libcrypto.so.10
 #4 0x00007ffb65589c58 in internal_verify () from /usr/lib64/libcrypto.so.10
 #5 0x00007ffb6558baef in X509_verify_cert () from /usr/lib64/libcrypto.so.10
 #6 0x00007ffb658d1417 in ssl_add_cert_chain () from /usr/lib64/libssl.so.10
 #7 0x00007ffb658b6095 in ssl3_output_cert_chain () from /usr/lib64/libssl.so.10
 #8 0x00007ffb658ae894 in ssl3_send_client_certificate () from /usr/lib64/libssl.so.10
 #9 0x00007ffb658aeecf in ssl3_connect () from /usr/lib64/libssl.so.10
 #10 0x00007ffb658b8e7e in ssl23_connect () from /usr/lib64/libssl.so.10


  I am setting context to use SSLv23_method() s. However, I can see ssl3_ methods being called. Is there any issue with that?  

Given below is SSL* object passed to SSL_do_handshake method.

(gdb) p *p_SSLCtx
$2 = {version = 771, type = 4096, method = 0x7ffb65af3320, rbio = 0x7ffb5819e140, wbio = 0x7ffb58193570, bbio = 0x7ffb58193570, rwstate = 1, in_handshake = 2, handshake_func = 0x7ffb658aea30 <ssl3_connect>, server = 0, new_session = 0, quiet_shutdown = 0, shutdown = 0, state = 4467,
  rstate = 240, init_buf = 0x7ffb581934b0, init_msg = 0x7ffb581e10d4, init_num = 0, init_off = 0, packet = 0x7ffb581e6633 "\026\003\003", packet_length = 0, s2 = 0x0, s3 = 0x7ffb58195ee0, d1 = 0x0, read_ahead = 0, msg_callback = 0x0, msg_callback_arg = 0x0, hit = 0, param = 0x7ffb581933f0,
  cipher_list = 0x0, cipher_list_by_id = 0x0, mac_flags = 0, enc_read_ctx = 0x0, read_hash = 0x0, expand = 0x0, enc_write_ctx = 0x0, write_hash = 0x0, compress = 0x0, cert = 0x7ffb58195ba0, sid_ctx_length = 0, sid_ctx = '\000' <repeats 31 times>, session = 0x7ffb58198100,
  generate_session_id = 0x0,......
 


--
Gimhani Uthpala Kankanamge

Reply | Threaded
Open this post in threaded view
|

RE: Random and rare Seg faults at openssl library level

Michael Wojcik
> From: openssl-users <[hidden email]> On Behalf Of Gimhani Uthpala
> Sent: Wednesday, 6 January, 2021 10:10

> I'm running an application which uses openssl for secure communication between
> processes. I am getting seg-faults at openssl level. This only occurred very
> randomly and the following are stacks that seg faults  at openssl level in the
> given 2 cases.

> We are using openssl 1.0.2k.

Sometimes you see a question that nearly answers itself.

You're using a release that's approaching four years old, and which is unsupported, unless you have a premium support contract from openssl.org or similar through another vendor. If you do, that's whom you should ask.

In any case, why are you using 1.0.2k? At the very least you should be using the final 1.0.2 release -- and then only if you absolutely can't move to 1.1.1 (generally because you need FIPS validation, but you don't mention FIPS). And then you need a premium support contract, if this is a commercial product. Particularly these days it's very hard to forgive a commercial-software vendor using an outdated, unsupported third-party component.

The most recent version of 1.0.2 that I happen to have lying around is 1.0.2n, and there's nothing in the changelog between 1.0.2k and 1.0.2n which looks likely to cause this particular problem (though CVE-2017-3735 is a slight contender). But that just means the cause isn't anything obvious between k and n.

> Went through the security vulnerabilities list for this version but couldn't
> find a clue. Running valgrind too didn't give an exact clue related to the issue.
> Can you please guide me how can I find the exact root cause for the seg fault?

The same way you'd track down an intermittent cause of Undefined Behavior in any other program: some combination of dynamic monitoring, symbolic execution, static code analysis, source code review, testing variants, tracing, fuzzing, post-mortem analysis, and so on. This isn't specific to OpenSSL.

But you're asking the wrong question. The correct question is: Why are you using an outdated version of OpenSSL?

--
Michael Wojcik
Reply | Threaded
Open this post in threaded view
|

Re: Random and rare Seg faults at openssl library level

tincanteksup


On 06/01/2021 20:57, Michael Wojcik wrote:

>
> But you're asking the wrong question. The correct question is: Why are you using an outdated version of OpenSSL?
>
> --
> Michael Wojcik
>

:whip-crack: !

Reply | Threaded
Open this post in threaded view
|

Re: Random and rare Seg faults at openssl library level

Ken Goldman-2
In reply to this post by Gimhani Uthpala
On 1/6/2021 12:10 PM, Gimhani Uthpala wrote:

> I am getting seg-faults at openssl level. This only occurred very randomly and the following are stacks that seg faults  at openssl level in the given 2 cases. We are using openssl 1.0.2k.

The usual cause is that you are compiling with one version of openssl and (static or dynamic) linking with a different one.
The cause of that is typically that you have more than one version of openssl installed.

If this is a 3rd party application, not one you're building, you have to find out what version of openssl they expect.


Reply | Threaded
Open this post in threaded view
|

Re: Random and rare Seg faults at openssl library level

Jan Just Keijser-2
In reply to this post by Gimhani Uthpala
Hi,

On 06/01/21 18:10, Gimhani Uthpala wrote:
Dear team,
I'm running an application which uses openssl for secure communication between processes. I am getting seg-faults at openssl level. This only occurred very randomly and the following are stacks that seg faults  at openssl level in the given 2 cases. We are using openssl 1.0.2k. 

version 1.0.2k suggests you are using RHEL7/CentOS 7, correct?
Went through the security vulnerabilities list for this version but couldn't find a clue. Running valgrind too didn't give an exact clue related to the issue. Can you please guide me how can I find the exact root cause for the seg fault?  

I am calling SSL_do_handshake(ssl_ctx) from my code level and both the below seg faults are occuring from it's inside. 

#0  0x00007fd64cdabdd3 in ASN1_item_verify () from /lib64/libcrypto.so.10
#1  0x00007fd64cdcac58 in internal_verify () from /lib64/libcrypto.so.10
#2  0x00007fd64cdccaef in X509_verify_cert () from /lib64/libcrypto.so.10
#3  0x00007fd64d111c68 in ssl_verify_cert_chain () from /lib64/libssl.so.10
#4  0x00007fd64d0e8cc6 in ssl3_get_client_certificate () from /lib64/libssl.so.10
#5  0x00007fd64d0ea3f8 in ssl3_accept () from /lib64/libssl.so.10


so the segfault occurs inside ASN1_item_verify () when verifying the certificate - it could be a malformed certificate with invalid ASN1 encoding; do you have the certificate that causes the segfault?

If you do not, then it is worthwhile recording/storing all certificates until you find the one that causes the segfault and then examine it.


HTH,

JJK
Reply | Threaded
Open this post in threaded view
|

Re: Random and rare Seg faults at openssl library level

Jan Just Keijser-2
In reply to this post by Michael Wojcik
On 06/01/21 21:57, Michael Wojcik wrote:
>
> The same way you'd track down an intermittent cause of Undefined Behavior in any other program: some combination of dynamic monitoring, symbolic execution, static code analysis, source code review, testing variants, tracing, fuzzing, post-mortem analysis, and so on. This isn't specific to OpenSSL.
>
> But you're asking the wrong question. The correct question is: Why are you using an outdated version of OpenSSL?

possibly because:

$ cat /etc/redhat-release && openssl version
CentOS Linux release 7.9.2009 (Core)
OpenSSL 1.0.2k-fips  26 Jan 2017

?


Reply | Threaded
Open this post in threaded view
|

Re: Random and rare Seg faults at openssl library level

John Wasilewski
Please remove my email

> On Jan 7, 2021, at 3:23 AM, Jan Just Keijser <[hidden email]> wrote:
>
> On 06/01/21 21:57, Michael Wojcik wrote:
>>
>> The same way you'd track down an intermittent cause of Undefined Behavior in any other program: some combination of dynamic monitoring, symbolic execution, static code analysis, source code review, testing variants, tracing, fuzzing, post-mortem analysis, and so on. This isn't specific to OpenSSL.
>>
>> But you're asking the wrong question. The correct question is: Why are you using an outdated version of OpenSSL?
>
> possibly because:
>
> $ cat /etc/redhat-release && openssl version
> CentOS Linux release 7.9.2009 (Core)
> OpenSSL 1.0.2k-fips  26 Jan 2017
>
> ?
>
>
Reply | Threaded
Open this post in threaded view
|

Re: Random and rare Seg faults at openssl library level

Matt Caswell-2
In reply to this post by Gimhani Uthpala


On 06/01/2021 17:10, Gimhani Uthpala wrote:
>   I am setting context to use SSLv23_method() s. However, I can see
> ssl3_ methods being called. Is there any issue with that?  

Just answering this one side question: no, this is normal behaviour.

Matt

Reply | Threaded
Open this post in threaded view
|

RE: Random and rare Seg faults at openssl library level

Michael Wojcik
In reply to this post by Jan Just Keijser-2
> From: Jan Just Keijser <[hidden email]>
> Sent: Thursday, 7 January, 2021 01:23
>
> On 06/01/21 21:57, Michael Wojcik wrote:
> >
> >
> > But you're asking the wrong question. The correct question is: Why are you
> > using an outdated version of OpenSSL?
>
> possibly because:
>
> $ cat /etc/redhat-release && openssl version
> CentOS Linux release 7.9.2009 (Core)
> OpenSSL 1.0.2k-fips  26 Jan 2017

Ugh. Well, OP should have made that clear in the original message.

And this is one of the problems with using an OpenSSL supplied by the OS vendor.

--
Michael Wojcik
Reply | Threaded
Open this post in threaded view
|

Re: Random and rare Seg faults at openssl library level

Ken Goldman-2
On 1/7/2021 10:11 AM, Michael Wojcik wrote:
>>
>> $ cat /etc/redhat-release && openssl version
>> CentOS Linux release 7.9.2009 (Core)
>> OpenSSL 1.0.2k-fips  26 Jan 2017
>
> Ugh. Well, OP should have made that clear in the original message.
>
> And this is one of the problems with using an OpenSSL supplied by the OS vendor.

In defense of "the OS vendor", meaning the distro, it's a big task to
upgrade to a new openssl major release.  Because there is often not ABI
compatibility, every package has to be ported, built, and tested.
A distro release that is in long term support doesn't do that often.


Reply | Threaded
Open this post in threaded view
|

Re: Random and rare Seg faults at openssl library level

Gimhani Uthpala
In reply to this post by Jan Just Keijser-2


On Thu, Jan 7, 2021 at 1:51 PM Jan Just Keijser <[hidden email]> wrote:
Hi,

On 06/01/21 18:10, Gimhani Uthpala wrote:
Dear team,
I'm running an application which uses openssl for secure communication between processes. I am getting seg-faults at openssl level. This only occurred very randomly and the following are stacks that seg faults  at openssl level in the given 2 cases. We are using openssl 1.0.2k. 

version 1.0.2k suggests you are using RHEL7/CentOS 7, correct?

Yes, I am using RHEL7 and using its openssl version 1.0.2k-fips.
 
Went through the security vulnerabilities list for this version but couldn't find a clue. Running valgrind too didn't give an exact clue related to the issue. Can you please guide me how can I find the exact root cause for the seg fault?  

I am calling SSL_do_handshake(ssl_ctx) from my code level and both the below seg faults are occuring from it's inside. 

#0  0x00007fd64cdabdd3 in ASN1_item_verify () from /lib64/libcrypto.so.10
#1  0x00007fd64cdcac58 in internal_verify () from /lib64/libcrypto.so.10
#2  0x00007fd64cdccaef in X509_verify_cert () from /lib64/libcrypto.so.10
#3  0x00007fd64d111c68 in ssl_verify_cert_chain () from /lib64/libssl.so.10
#4  0x00007fd64d0e8cc6 in ssl3_get_client_certificate () from /lib64/libssl.so.10
#5  0x00007fd64d0ea3f8 in ssl3_accept () from /lib64/libssl.so.10


so the segfault occurs inside ASN1_item_verify () when verifying the certificate - it could be a malformed certificate with invalid ASN1 encoding; do you have the certificate that causes the segfault? 

If you do not, then it is worthwhile recording/storing all certificates until you find the one that causes the segfault and then examine it.
  
I do not have access to the certificate that caused segfault. Will try to record all certs to check this, Thanks.  
 


HTH,

JJK



Reply | Threaded
Open this post in threaded view
|

Re: Random and rare Seg faults at openssl library level

Gimhani Uthpala
In reply to this post by Jan Just Keijser-2

On Thu, Jan 7, 2021 at 1:53 PM Jan Just Keijser <[hidden email]> wrote:
On 06/01/21 21:57, Michael Wojcik wrote:
>
> The same way you'd track down an intermittent cause of Undefined Behavior in any other program: some combination of dynamic monitoring, symbolic execution, static code analysis, source code review, testing variants, tracing, fuzzing, post-mortem analysis, and so on. This isn't specific to OpenSSL.
>
> But you're asking the wrong question. The correct question is: Why are you using an outdated version of OpenSSL?

possibly because:

$ cat /etc/redhat-release && openssl version
CentOS Linux release 7.9.2009 (Core)
OpenSSL 1.0.2k-fips  26 Jan 2017

?


  
Yes, using this openssl version coming with the OS.  


Reply | Threaded
Open this post in threaded view
|

Re: Random and rare Seg faults at openssl library level

Gimhani Uthpala
In reply to this post by Ken Goldman-2


On Thu, Jan 7, 2021 at 3:08 AM Ken Goldman <[hidden email]> wrote:
On 1/6/2021 12:10 PM, Gimhani Uthpala wrote:

> I am getting seg-faults at openssl level. This only occurred very randomly and the following are stacks that seg faults  at openssl level in the given 2 cases. We are using openssl 1.0.2k.

The usual cause is that you are compiling with one version of openssl and (static or dynamic) linking with a different one.
The cause of that is typically that you have more than one version of openssl installed.

If this is a 3rd party application, not one you're building, you have to find out what version of openssl they expect.



I only have this 1.0.2.k-fips one version installed in both compiling and running machines. However, I am compiling the application in RH7.4 and running in RH7.8 linking to openssl library dynamically. I assume no issue with that as I am using the same version of openssl in both.

-Gimhani
Reply | Threaded
Open this post in threaded view
|

RE: Random and rare Seg faults at openssl library level

Ken Goldman-2

Inactive hide details for Gimhani Uthpala ---01/07/2021 05:53:32 PM---On Thu, Jan 7, 2021 at 3:08 AM Ken Goldman <kgoldman@us.iGimhani Uthpala ---01/07/2021 05:53:32 PM---On Thu, Jan 7, 2021 at 3:08 AM Ken Goldman <[hidden email]> wrote: > On 1/6/2021 12:10 PM, Gimh

From: Gimhani Uthpala <[hidden email]>
To: Ken Goldman <[hidden email]>
Cc: [hidden email]
Date: 01/07/2021 05:53 PM
Subject: [EXTERNAL] Re: Random and rare Seg faults at openssl library level





I only have this 1.0.2.k-fips one version installed in both compiling and running machines. However, I am compiling the application in RH7.4 and running in RH7.8 linking to openssl library dynamically. I assume no issue with that as I am using the same version of openssl in both.

You are having a problem, and that is a typical cause. Try compiling and running on the exact same OS.

If you installed openssl yourself, not using the RHEL yum installer, I would expect random and rare issues.
Reply | Threaded
Open this post in threaded view
|

Re: Random and rare Seg faults at openssl library level

OpenSSL - User mailing list
In reply to this post by Ken Goldman-2
On 2021-01-07 18:05, Ken Goldman wrote:

> On 1/7/2021 10:11 AM, Michael Wojcik wrote:
>>>
>>> $ cat /etc/redhat-release && openssl version
>>> CentOS Linux release 7.9.2009 (Core)
>>> OpenSSL 1.0.2k-fips  26 Jan 2017
>>
>> Ugh. Well, OP should have made that clear in the original message.
>>
>> And this is one of the problems with using an OpenSSL supplied by the
>> OS vendor.
>
> In defense of "the OS vendor", meaning the distro, it's a big task to
> upgrade to a new openssl major release.  Because there is often not ABI
> compatibility, every package has to be ported, built, and tested.
> A distro release that is in long term support doesn't do that often.
>
>

In defense of long term support distros, until a few years ago, no one
suspected that OpenSSL would come under a new leadership that actively
did everything to make it near-impossible to maintain backported
security patches for a typical 5+ year distro lifecycle (with
OpenSSL-independent start date).

Until 1.0.2, all OpenSSL releases were incremental patch-steps from the
old 0.9.x series, allowing distro maintainers to manually cherry pick
changes for doing ABI-compatible patches for whichever 1.0.x or 0.9.x
was current at the start of their lifecycle.  Then the new leadership
started to restructure the code even in supposedly patch-level releases.

A lot of long term support distros are now firmly stuck with unsupported
OpenSSL 1.0.2 and/or short life cycle 1.1.1.

Not all long term distros are run by rich companies like IBM/RedHat that
can purchase support plans, resulting in further popularity of OpenSSL
forks.

Enjoy

Jakob
--
Jakob Bohm, CIO, Partner, WiseMo A/S.  https://www.wisemo.com
Transformervej 29, 2860 Søborg, Denmark.  Direct +45 31 13 16 10
This public discussion message is non-binding and may contain errors.
WiseMo - Remote Service Management for PCs, Phones and Embedded

Reply | Threaded
Open this post in threaded view
|

Re: Random and rare Seg faults at openssl library level

Gimhani Uthpala
In reply to this post by Gimhani Uthpala
Hi team,
https://www.openssl.org/docs/man1.0.2/man3/CRYPTO_set_locking_callback.html :  From this , learnt that for openssl multi-threaded applications to be run safely, the callback functions to be implemented.

I am using this in a multi-threaded application and Above scenario was again recreated and in that case 2 threads were calling RSA_verify (same stack above). However, for SSL_do_handshake, SSL* objects are given separately and they are not shared between threads. So, we have not implemented callback functions.

Could you please clarify whether we need to implement locking callbacks for multi-threaded applications  even if the ssl objects we supply are provided separately for threads?

Many thanks.

On Wed, Jan 6, 2021 at 10:40 PM Gimhani Uthpala <[hidden email]> wrote:
Dear team,
I'm running an application which uses openssl for secure communication between processes. I am getting seg-faults at openssl level. This only occurred very randomly and the following are stacks that seg faults  at openssl level in the given 2 cases. We are using openssl 1.0.2k. 

Went through the security vulnerabilities list for this version but couldn't find a clue. Running valgrind too didn't give an exact clue related to the issue. Can you please guide me how can I find the exact root cause for the seg fault?  

I am calling SSL_do_handshake(ssl_ctx) from my code level and both the below seg faults are occuring from it's inside. 

#0  0x00007fd64cdabdd3 in ASN1_item_verify () from /lib64/libcrypto.so.10
#1  0x00007fd64cdcac58 in internal_verify () from /lib64/libcrypto.so.10
#2  0x00007fd64cdccaef in X509_verify_cert () from /lib64/libcrypto.so.10
#3  0x00007fd64d111c68 in ssl_verify_cert_chain () from /lib64/libssl.so.10
#4  0x00007fd64d0e8cc6 in ssl3_get_client_certificate () from /lib64/libssl.so.10
#5  0x00007fd64d0ea3f8 in ssl3_accept () from /lib64/libssl.so.10


 #0 0x00007ffb65529854 in RSA_verify () from /usr/lib64/libcrypto.so.10
 #1 0x00007ffb6552f1fc in pkey_rsa_verify () from /usr/lib64/libcrypto.so.10
 #2 0x00007ffb65561877 in EVP_DigestVerifyFinal () from /usr/lib64/libcrypto.so.10
 #3 0x00007ffb6556af25 in ASN1_item_verify () from /usr/lib64/libcrypto.so.10
 #4 0x00007ffb65589c58 in internal_verify () from /usr/lib64/libcrypto.so.10
 #5 0x00007ffb6558baef in X509_verify_cert () from /usr/lib64/libcrypto.so.10
 #6 0x00007ffb658d1417 in ssl_add_cert_chain () from /usr/lib64/libssl.so.10
 #7 0x00007ffb658b6095 in ssl3_output_cert_chain () from /usr/lib64/libssl.so.10
 #8 0x00007ffb658ae894 in ssl3_send_client_certificate () from /usr/lib64/libssl.so.10
 #9 0x00007ffb658aeecf in ssl3_connect () from /usr/lib64/libssl.so.10
 #10 0x00007ffb658b8e7e in ssl23_connect () from /usr/lib64/libssl.so.10


  I am setting context to use SSLv23_method() s. However, I can see ssl3_ methods being called. Is there any issue with that?  

Given below is SSL* object passed to SSL_do_handshake method.

(gdb) p *p_SSLCtx
$2 = {version = 771, type = 4096, method = 0x7ffb65af3320, rbio = 0x7ffb5819e140, wbio = 0x7ffb58193570, bbio = 0x7ffb58193570, rwstate = 1, in_handshake = 2, handshake_func = 0x7ffb658aea30 <ssl3_connect>, server = 0, new_session = 0, quiet_shutdown = 0, shutdown = 0, state = 4467,
  rstate = 240, init_buf = 0x7ffb581934b0, init_msg = 0x7ffb581e10d4, init_num = 0, init_off = 0, packet = 0x7ffb581e6633 "\026\003\003", packet_length = 0, s2 = 0x0, s3 = 0x7ffb58195ee0, d1 = 0x0, read_ahead = 0, msg_callback = 0x0, msg_callback_arg = 0x0, hit = 0, param = 0x7ffb581933f0,
  cipher_list = 0x0, cipher_list_by_id = 0x0, mac_flags = 0, enc_read_ctx = 0x0, read_hash = 0x0, expand = 0x0, enc_write_ctx = 0x0, write_hash = 0x0, compress = 0x0, cert = 0x7ffb58195ba0, sid_ctx_length = 0, sid_ctx = '\000' <repeats 31 times>, session = 0x7ffb58198100,
  generate_session_id = 0x0,......
 


--
Gimhani Uthpala Kankanamge



--
Gimhani Uthpala Kankanamge
Reply | Threaded
Open this post in threaded view
|

Re: Random and rare Seg faults at openssl library level

Jan Just Keijser-2
In reply to this post by Gimhani Uthpala
Hi,

On 07/01/21 23:53, Gimhani Uthpala wrote:


On Thu, Jan 7, 2021 at 3:08 AM Ken Goldman <[hidden email]> wrote:
On 1/6/2021 12:10 PM, Gimhani Uthpala wrote:

> I am getting seg-faults at openssl level. This only occurred very randomly and the following are stacks that seg faults  at openssl level in the given 2 cases. We are using openssl 1.0.2k.

The usual cause is that you are compiling with one version of openssl and (static or dynamic) linking with a different one.
The cause of that is typically that you have more than one version of openssl installed.

If this is a 3rd party application, not one you're building, you have to find out what version of openssl they expect.



I only have this 1.0.2.k-fips one version installed in both compiling and running machines. However, I am compiling the application in RH7.4 and running in RH7.8 linking to openssl library dynamically. I assume no issue with that as I am using the same version of openssl in both.

actually - there could be an issue with that, as RedHat has this tendency to patch openssl between releases (mostly backporting security fixes from openssl 1.1.x to 1.0.2k).

Have you tried installing the debuginfo package for openssl so that the stacktrace will show you better info:
  debuginfo-install openssl-libs openssl

right now all you know is that the segfault occurs *somewhere* within ASN1_item_verify ()

HTH,

JJK

Reply | Threaded
Open this post in threaded view
|

Re: Random and rare Seg faults at openssl library level

Matt Caswell-2
In reply to this post by Gimhani Uthpala


On 12/01/2021 04:23, Gimhani Uthpala wrote:

> Hi team,
> https://www.openssl.org/docs/man1.0.2/man3/CRYPTO_set_locking_callback.html
> :  From this , learnt that for openssl multi-threaded applications to be
> run safely, the callback functions to be implemented.
>
> I am using this in a multi-threaded application and Above scenario was
> again recreated and in that case 2 threads were calling RSA_verify (same
> stack above). However, for SSL_do_handshake, SSL* objects are given
> separately and they are not shared between threads. So, we have not
> implemented callback functions.
>
> Could you please clarify whether we need to implement locking callbacks
> for multi-threaded applications  even if the ssl objects we supply are
> provided separately for threads?

If you are using 1.0.2 in a multi-threaded context then you *must*
implement the locking callbacks. There are resources that are shared
between multiple threads that require appropriate locking.

Matt



>
> Many thanks.
>
> On Wed, Jan 6, 2021 at 10:40 PM Gimhani Uthpala
> <[hidden email] <mailto:[hidden email]>> wrote:
>
>     Dear team,
>     I'm running an application which uses openssl for secure
>     communication between processes. I am getting seg-faults at openssl
>     level. This only occurred very randomly and the following are stacks
>     that seg faults  at openssl level in the given 2 cases. We are using
>     openssl 1.0.2k. 
>
>     Went through the security vulnerabilities list for this version but
>     couldn't find a clue. Running valgrind too didn't give an exact clue
>     related to the issue. Can you please guide me how can I find the
>     exact root cause for the seg fault?  
>
>     I am calling SSL_do_handshake(ssl_ctx) from my code level and both
>     the below seg faults are occuring from it's inside. 
>
>     #0  0x00007fd64cdabdd3 in ASN1_item_verify () from
>     /lib64/libcrypto.so.10
>     #1  0x00007fd64cdcac58 in internal_verify () from /lib64/libcrypto.so.10
>     #2  0x00007fd64cdccaef in X509_verify_cert () from
>     /lib64/libcrypto.so.10
>     #3  0x00007fd64d111c68 in ssl_verify_cert_chain () from
>     /lib64/libssl.so.10
>     #4  0x00007fd64d0e8cc6 in ssl3_get_client_certificate () from
>     /lib64/libssl.so.10
>     *#5  0x00007fd64d0ea3f8 in ssl3_accept () from /lib64/libssl.so.10*
>
>
>      #0 0x00007ffb65529854 in RSA_verify () from /usr/lib64/libcrypto.so.10
>      #1 0x00007ffb6552f1fc in pkey_rsa_verify () from
>     /usr/lib64/libcrypto.so.10
>      #2 0x00007ffb65561877 in EVP_DigestVerifyFinal () from
>     /usr/lib64/libcrypto.so.10
>      #3 0x00007ffb6556af25 in ASN1_item_verify () from
>     /usr/lib64/libcrypto.so.10
>      #4 0x00007ffb65589c58 in internal_verify () from
>     /usr/lib64/libcrypto.so.10
>      #5 0x00007ffb6558baef in X509_verify_cert () from
>     /usr/lib64/libcrypto.so.10
>      #6 0x00007ffb658d1417 in ssl_add_cert_chain () from
>     /usr/lib64/libssl.so.10
>      #7 0x00007ffb658b6095 in ssl3_output_cert_chain () from
>     /usr/lib64/libssl.so.10
>      #8 0x00007ffb658ae894 in ssl3_send_client_certificate () from
>     /usr/lib64/libssl.so.10
>     * #9 0x00007ffb658aeecf in ssl3_connect () from /usr/lib64/libssl.so.10
>      #10 0x00007ffb658b8e7e in ssl23_connect () from
>     /usr/lib64/libssl.so.10*
>
>       I am setting context to use SSLv23_method() s. However, I can see
>     ssl3_ methods being called. Is there any issue with that?  
>
>     Given below is SSL* object passed to SSL_do_handshake method.
>
>     (gdb) p *p_SSLCtx
>     $2 = {version = 771, type = 4096, method = 0x7ffb65af3320, rbio =
>     0x7ffb5819e140, wbio = 0x7ffb58193570, bbio = 0x7ffb58193570,
>     rwstate = 1, in_handshake = 2, *handshake_func = 0x7ffb658aea30
>     <ssl3_connect>*, server = 0, new_session = 0, quiet_shutdown = 0,
>     shutdown = 0, state = 4467,
>       rstate = 240, init_buf = 0x7ffb581934b0, init_msg =
>     0x7ffb581e10d4, init_num = 0, init_off = 0, *packet = 0x7ffb581e6633
>     "\026\003\003", packet_length = 0*, s2 = 0x0, s3 = 0x7ffb58195ee0,
>     d1 = 0x0, read_ahead = 0, msg_callback = 0x0, msg_callback_arg =
>     0x0, hit = 0, param = 0x7ffb581933f0,
>       cipher_list = 0x0, cipher_list_by_id = 0x0, mac_flags = 0,
>     enc_read_ctx = 0x0, read_hash = 0x0, expand = 0x0, enc_write_ctx =
>     0x0, write_hash = 0x0, compress = 0x0, cert = 0x7ffb58195ba0,
>     sid_ctx_length = 0, sid_ctx = '\000' <repeats 31 times>, session =
>     0x7ffb58198100,
>       generate_session_id = 0x0,......
>      
>
>
>     --
>     *Gimhani Uthpala Kankanamge*
>
>
>
> --
> *Gimhani Uthpala Kankanamge*