Random and rare Seg faults at openssl library level
Locked
Random and rare Seg faults at openssl library level
 
|
Dear team,
I'm running an application which uses openssl for secure communication between processes. I am getting seg-faults at openssl level. This only occurred very randomly and the following are stacks that seg faults at openssl level in the given 2 cases. We are using openssl 1.0.2k. Went through the security vulnerabilities list for this version but couldn't find a clue. Running valgrind too didn't give an exact clue related to the issue. Can you please guide me how can I find the exact root cause for the seg fault? I am calling SSL_do_handshake(ssl_ctx) from my code level and both the below seg faults are occuring from it's inside. #0 0x00007fd64cdabdd3 in ASN1_item_verify () from /lib64/libcrypto.so.10 #1 0x00007fd64cdcac58 in internal_verify () from /lib64/libcrypto.so.10 #2 0x00007fd64cdccaef in X509_verify_cert () from /lib64/libcrypto.so.10 #3 0x00007fd64d111c68 in ssl_verify_cert_chain () from /lib64/libssl.so.10 #4 0x00007fd64d0e8cc6 in ssl3_get_client_certificate () from /lib64/libssl.so.10 #5 0x00007fd64d0ea3f8 in ssl3_accept () from /lib64/libssl.so.10 #0 0x00007ffb65529854 in RSA_verify () from /usr/lib64/libcrypto.so.10 #1 0x00007ffb6552f1fc in pkey_rsa_verify () from /usr/lib64/libcrypto.so.10 #2 0x00007ffb65561877 in EVP_DigestVerifyFinal () from /usr/lib64/libcrypto.so.10 #3 0x00007ffb6556af25 in ASN1_item_verify () from /usr/lib64/libcrypto.so.10 #4 0x00007ffb65589c58 in internal_verify () from /usr/lib64/libcrypto.so.10 #5 0x00007ffb6558baef in X509_verify_cert () from /usr/lib64/libcrypto.so.10 #6 0x00007ffb658d1417 in ssl_add_cert_chain () from /usr/lib64/libssl.so.10 #7 0x00007ffb658b6095 in ssl3_output_cert_chain () from /usr/lib64/libssl.so.10 #8 0x00007ffb658ae894 in ssl3_send_client_certificate () from /usr/lib64/libssl.so.10 #9 0x00007ffb658aeecf in ssl3_connect () from /usr/lib64/libssl.so.10 #10 0x00007ffb658b8e7e in ssl23_connect () from /usr/lib64/libssl.so.10 I am setting context to use SSLv23_method() s. However, I can see ssl3_ methods being called. Is there any issue with that? Given below is SSL* object passed to SSL_do_handshake method. $2 = {version = 771, type = 4096, method = 0x7ffb65af3320, rbio = 0x7ffb5819e140, wbio = 0x7ffb58193570, bbio = 0x7ffb58193570, rwstate = 1, in_handshake = 2, handshake_func = 0x7ffb658aea30 <ssl3_connect>, server = 0, new_session = 0, quiet_shutdown = 0, shutdown = 0, state = 4467, rstate = 240, init_buf = 0x7ffb581934b0, init_msg = 0x7ffb581e10d4, init_num = 0, init_off = 0, packet = 0x7ffb581e6633 "\026\003\003", packet_length = 0, s2 = 0x0, s3 = 0x7ffb58195ee0, d1 = 0x0, read_ahead = 0, msg_callback = 0x0, msg_callback_arg = 0x0, hit = 0, param = 0x7ffb581933f0, cipher_list = 0x0, cipher_list_by_id = 0x0, mac_flags = 0, enc_read_ctx = 0x0, read_hash = 0x0, expand = 0x0, enc_write_ctx = 0x0, write_hash = 0x0, compress = 0x0, cert = 0x7ffb58195ba0, sid_ctx_length = 0, sid_ctx = '\000' <repeats 31 times>, session = 0x7ffb58198100, generate_session_id = 0x0,...... Gimhani Uthpala Kankanamge |
RE: Random and rare Seg faults at openssl library level
|
|
> From: openssl-users <[hidden email]> On Behalf Of Gimhani Uthpala
> Sent: Wednesday, 6 January, 2021 10:10 > I'm running an application which uses openssl for secure communication between > processes. I am getting seg-faults at openssl level. This only occurred very > randomly and the following are stacks that seg faults at openssl level in the > given 2 cases. > We are using openssl 1.0.2k. Sometimes you see a question that nearly answers itself. You're using a release that's approaching four years old, and which is unsupported, unless you have a premium support contract from openssl.org or similar through another vendor. If you do, that's whom you should ask. In any case, why are you using 1.0.2k? At the very least you should be using the final 1.0.2 release -- and then only if you absolutely can't move to 1.1.1 (generally because you need FIPS validation, but you don't mention FIPS). And then you need a premium support contract, if this is a commercial product. Particularly these days it's very hard to forgive a commercial-software vendor using an outdated, unsupported third-party component. The most recent version of 1.0.2 that I happen to have lying around is 1.0.2n, and there's nothing in the changelog between 1.0.2k and 1.0.2n which looks likely to cause this particular problem (though CVE-2017-3735 is a slight contender). But that just means the cause isn't anything obvious between k and n. > Went through the security vulnerabilities list for this version but couldn't > find a clue. Running valgrind too didn't give an exact clue related to the issue. > Can you please guide me how can I find the exact root cause for the seg fault? The same way you'd track down an intermittent cause of Undefined Behavior in any other program: some combination of dynamic monitoring, symbolic execution, static code analysis, source code review, testing variants, tracing, fuzzing, post-mortem analysis, and so on. This isn't specific to OpenSSL. But you're asking the wrong question. The correct question is: Why are you using an outdated version of OpenSSL? -- Michael Wojcik |
Re: Random and rare Seg faults at openssl library level
|
|
On 06/01/2021 20:57, Michael Wojcik wrote: > > But you're asking the wrong question. The correct question is: Why are you using an outdated version of OpenSSL? > > -- > Michael Wojcik > :whip-crack: ! |
Re: Random and rare Seg faults at openssl library level
|
|
In reply to this post by Gimhani Uthpala
On 1/6/2021 12:10 PM, Gimhani Uthpala wrote:
> I am getting seg-faults at openssl level. This only occurred very randomly and the following are stacks that seg faults at openssl level in the given 2 cases. We are using openssl 1.0.2k. The usual cause is that you are compiling with one version of openssl and (static or dynamic) linking with a different one. The cause of that is typically that you have more than one version of openssl installed. If this is a 3rd party application, not one you're building, you have to find out what version of openssl they expect. |
Re: Random and rare Seg faults at openssl library level
|
|
In reply to this post by Gimhani Uthpala
Hi,
On 06/01/21 18:10, Gimhani Uthpala wrote: version 1.0.2k suggests you are using RHEL7/CentOS 7, correct? so the segfault occurs inside ASN1_item_verify () when verifying the certificate - it could be a malformed certificate with invalid ASN1 encoding; do you have the certificate that causes the segfault? If you do not, then it is worthwhile recording/storing all certificates until you find the one that causes the segfault and then examine it. HTH, JJK |
Re: Random and rare Seg faults at openssl library level
|
|
In reply to this post by Michael Wojcik
On 06/01/21 21:57, Michael Wojcik wrote:
> > The same way you'd track down an intermittent cause of Undefined Behavior in any other program: some combination of dynamic monitoring, symbolic execution, static code analysis, source code review, testing variants, tracing, fuzzing, post-mortem analysis, and so on. This isn't specific to OpenSSL. > > But you're asking the wrong question. The correct question is: Why are you using an outdated version of OpenSSL? possibly because: $ cat /etc/redhat-release && openssl version CentOS Linux release 7.9.2009 (Core) OpenSSL 1.0.2k-fips 26 Jan 2017 ? |
Re: Random and rare Seg faults at openssl library level
|
|
Please remove my email
> On Jan 7, 2021, at 3:23 AM, Jan Just Keijser <[hidden email]> wrote: > > On 06/01/21 21:57, Michael Wojcik wrote: >> >> The same way you'd track down an intermittent cause of Undefined Behavior in any other program: some combination of dynamic monitoring, symbolic execution, static code analysis, source code review, testing variants, tracing, fuzzing, post-mortem analysis, and so on. This isn't specific to OpenSSL. >> >> But you're asking the wrong question. The correct question is: Why are you using an outdated version of OpenSSL? > > possibly because: > > $ cat /etc/redhat-release && openssl version > CentOS Linux release 7.9.2009 (Core) > OpenSSL 1.0.2k-fips 26 Jan 2017 > > ? > > |
Re: Random and rare Seg faults at openssl library level
|
|
In reply to this post by Gimhani Uthpala
On 06/01/2021 17:10, Gimhani Uthpala wrote: > I am setting context to use SSLv23_method() s. However, I can see > ssl3_ methods being called. Is there any issue with that? Just answering this one side question: no, this is normal behaviour. Matt |
RE: Random and rare Seg faults at openssl library level
|
|
In reply to this post by Jan Just Keijser-2
> From: Jan Just Keijser <[hidden email]>
> Sent: Thursday, 7 January, 2021 01:23 > > On 06/01/21 21:57, Michael Wojcik wrote: > > > > > > But you're asking the wrong question. The correct question is: Why are you > > using an outdated version of OpenSSL? > > possibly because: > > $ cat /etc/redhat-release && openssl version > CentOS Linux release 7.9.2009 (Core) > OpenSSL 1.0.2k-fips 26 Jan 2017 Ugh. Well, OP should have made that clear in the original message. And this is one of the problems with using an OpenSSL supplied by the OS vendor. -- Michael Wojcik |
Re: Random and rare Seg faults at openssl library level
|
|
On 1/7/2021 10:11 AM, Michael Wojcik wrote:
>> >> $ cat /etc/redhat-release && openssl version >> CentOS Linux release 7.9.2009 (Core) >> OpenSSL 1.0.2k-fips 26 Jan 2017 > > Ugh. Well, OP should have made that clear in the original message. > > And this is one of the problems with using an OpenSSL supplied by the OS vendor. In defense of "the OS vendor", meaning the distro, it's a big task to upgrade to a new openssl major release. Because there is often not ABI compatibility, every package has to be ported, built, and tested. A distro release that is in long term support doesn't do that often. |
Re: Random and rare Seg faults at openssl library level
|
|
In reply to this post by Jan Just Keijser-2
On Thu, Jan 7, 2021 at 1:51 PM Jan Just Keijser <[hidden email]> wrote:
Yes, I am using RHEL7 and using its openssl version 1.0.2k-fips.
I do not have access to the certificate that caused segfault. Will try to record all certs to check this, Thanks.
|
Re: Random and rare Seg faults at openssl library level
|
|
In reply to this post by Jan Just Keijser-2
On Thu, Jan 7, 2021 at 1:53 PM Jan Just Keijser <[hidden email]> wrote: On 06/01/21 21:57, Michael Wojcik wrote: Yes, using this openssl version coming with the OS. |
Re: Random and rare Seg faults at openssl library level
|
|
In reply to this post by Ken Goldman-2
On Thu, Jan 7, 2021 at 3:08 AM Ken Goldman <[hidden email]> wrote: On 1/6/2021 12:10 PM, Gimhani Uthpala wrote: I only have this 1.0.2.k-fips one version installed in both compiling and running machines. However, I am compiling the application in RH7.4 and running in RH7.8 linking to openssl library dynamically. I assume no issue with that as I am using the same version of openssl in both. -Gimhani |
RE: Random and rare Seg faults at openssl library level
|
|
I only have this 1.0.2.k-fips one version installed in both compiling and running machines. However, I am compiling the application in RH7.4 and running in RH7.8 linking to openssl library dynamically. I assume no issue with that as I am using the same version of openssl in both. You are having a problem, and that is a typical cause. Try compiling and running on the exact same OS. If you installed openssl yourself, not using the RHEL yum installer, I would expect random and rare issues. |
Gimhani Uthpala ---01/07/2021 05:53:32 PM---On Thu, Jan 7, 2021 at 3:08 AM Ken Goldman <
Re: Random and rare Seg faults at openssl library level
|
|
In reply to this post by Ken Goldman-2
On 2021-01-07 18:05, Ken Goldman wrote:
> On 1/7/2021 10:11 AM, Michael Wojcik wrote: >>> >>> $ cat /etc/redhat-release && openssl version >>> CentOS Linux release 7.9.2009 (Core) >>> OpenSSL 1.0.2k-fips 26 Jan 2017 >> >> Ugh. Well, OP should have made that clear in the original message. >> >> And this is one of the problems with using an OpenSSL supplied by the >> OS vendor. > > In defense of "the OS vendor", meaning the distro, it's a big task to > upgrade to a new openssl major release. Because there is often not ABI > compatibility, every package has to be ported, built, and tested. > A distro release that is in long term support doesn't do that often. > > In defense of long term support distros, until a few years ago, no one suspected that OpenSSL would come under a new leadership that actively did everything to make it near-impossible to maintain backported security patches for a typical 5+ year distro lifecycle (with OpenSSL-independent start date). Until 1.0.2, all OpenSSL releases were incremental patch-steps from the old 0.9.x series, allowing distro maintainers to manually cherry pick changes for doing ABI-compatible patches for whichever 1.0.x or 0.9.x was current at the start of their lifecycle. Then the new leadership started to restructure the code even in supposedly patch-level releases. A lot of long term support distros are now firmly stuck with unsupported OpenSSL 1.0.2 and/or short life cycle 1.1.1. Not all long term distros are run by rich companies like IBM/RedHat that can purchase support plans, resulting in further popularity of OpenSSL forks. Enjoy Jakob -- Jakob Bohm, CIO, Partner, WiseMo A/S. https://www.wisemo.com Transformervej 29, 2860 Søborg, Denmark. Direct +45 31 13 16 10 This public discussion message is non-binding and may contain errors. WiseMo - Remote Service Management for PCs, Phones and Embedded |
Re: Random and rare Seg faults at openssl library level
|
|
In reply to this post by Gimhani Uthpala
Hi team, https://www.openssl.org/docs/man1.0.2/man3/CRYPTO_set_locking_callback.html : From this , learnt that for openssl multi-threaded applications to be run safely, the callback functions to be implemented.
I am using this in a multi-threaded application and Above scenario was again recreated and in that case 2 threads were calling RSA_verify (same stack above). However, for SSL_do_handshake, SSL* objects are given separately and they are not shared between threads. So, we have not implemented callback functions. Could you please clarify whether we need to implement locking callbacks for multi-threaded applications even if the ssl objects we supply are provided separately for threads? Many thanks. On Wed, Jan 6, 2021 at 10:40 PM Gimhani Uthpala <[hidden email]> wrote:
Gimhani Uthpala Kankanamge |
Re: Random and rare Seg faults at openssl library level
|
|
In reply to this post by Gimhani Uthpala
Hi,
On 07/01/21 23:53, Gimhani Uthpala wrote: actually - there could be an issue with that, as RedHat has this tendency to patch openssl between releases (mostly backporting security fixes from openssl 1.1.x to 1.0.2k). Have you tried installing the debuginfo package for openssl so that the stacktrace will show you better info: debuginfo-install openssl-libs openssl right now all you know is that the segfault occurs *somewhere* within ASN1_item_verify () HTH, JJK |
Re: Random and rare Seg faults at openssl library level
|
|
In reply to this post by Gimhani Uthpala
On 12/01/2021 04:23, Gimhani Uthpala wrote: > Hi team, > https://www.openssl.org/docs/man1.0.2/man3/CRYPTO_set_locking_callback.html > : From this , learnt that for openssl multi-threaded applications to be > run safely, the callback functions to be implemented. > > I am using this in a multi-threaded application and Above scenario was > again recreated and in that case 2 threads were calling RSA_verify (same > stack above). However, for SSL_do_handshake, SSL* objects are given > separately and they are not shared between threads. So, we have not > implemented callback functions. > > Could you please clarify whether we need to implement locking callbacks > for multi-threaded applications even if the ssl objects we supply are > provided separately for threads? If you are using 1.0.2 in a multi-threaded context then you *must* implement the locking callbacks. There are resources that are shared between multiple threads that require appropriate locking. Matt > > Many thanks. > > On Wed, Jan 6, 2021 at 10:40 PM Gimhani Uthpala > <[hidden email] <mailto:[hidden email]>> wrote: > > Dear team, > I'm running an application which uses openssl for secure > communication between processes. I am getting seg-faults at openssl > level. This only occurred very randomly and the following are stacks > that seg faults at openssl level in the given 2 cases. We are using > openssl 1.0.2k. > > Went through the security vulnerabilities list for this version but > couldn't find a clue. Running valgrind too didn't give an exact clue > related to the issue. Can you please guide me how can I find the > exact root cause for the seg fault? > > I am calling SSL_do_handshake(ssl_ctx) from my code level and both > the below seg faults are occuring from it's inside. > > #0 0x00007fd64cdabdd3 in ASN1_item_verify () from > /lib64/libcrypto.so.10 > #1 0x00007fd64cdcac58 in internal_verify () from /lib64/libcrypto.so.10 > #2 0x00007fd64cdccaef in X509_verify_cert () from > /lib64/libcrypto.so.10 > #3 0x00007fd64d111c68 in ssl_verify_cert_chain () from > /lib64/libssl.so.10 > #4 0x00007fd64d0e8cc6 in ssl3_get_client_certificate () from > /lib64/libssl.so.10 > *#5 0x00007fd64d0ea3f8 in ssl3_accept () from /lib64/libssl.so.10* > > > #0 0x00007ffb65529854 in RSA_verify () from /usr/lib64/libcrypto.so.10 > #1 0x00007ffb6552f1fc in pkey_rsa_verify () from > /usr/lib64/libcrypto.so.10 > #2 0x00007ffb65561877 in EVP_DigestVerifyFinal () from > /usr/lib64/libcrypto.so.10 > #3 0x00007ffb6556af25 in ASN1_item_verify () from > /usr/lib64/libcrypto.so.10 > #4 0x00007ffb65589c58 in internal_verify () from > /usr/lib64/libcrypto.so.10 > #5 0x00007ffb6558baef in X509_verify_cert () from > /usr/lib64/libcrypto.so.10 > #6 0x00007ffb658d1417 in ssl_add_cert_chain () from > /usr/lib64/libssl.so.10 > #7 0x00007ffb658b6095 in ssl3_output_cert_chain () from > /usr/lib64/libssl.so.10 > #8 0x00007ffb658ae894 in ssl3_send_client_certificate () from > /usr/lib64/libssl.so.10 > * #9 0x00007ffb658aeecf in ssl3_connect () from /usr/lib64/libssl.so.10 > #10 0x00007ffb658b8e7e in ssl23_connect () from > /usr/lib64/libssl.so.10* > > I am setting context to use SSLv23_method() s. However, I can see > ssl3_ methods being called. Is there any issue with that? > > Given below is SSL* object passed to SSL_do_handshake method. > > (gdb) p *p_SSLCtx > $2 = {version = 771, type = 4096, method = 0x7ffb65af3320, rbio = > 0x7ffb5819e140, wbio = 0x7ffb58193570, bbio = 0x7ffb58193570, > rwstate = 1, in_handshake = 2, *handshake_func = 0x7ffb658aea30 > <ssl3_connect>*, server = 0, new_session = 0, quiet_shutdown = 0, > shutdown = 0, state = 4467, > rstate = 240, init_buf = 0x7ffb581934b0, init_msg = > 0x7ffb581e10d4, init_num = 0, init_off = 0, *packet = 0x7ffb581e6633 > "\026\003\003", packet_length = 0*, s2 = 0x0, s3 = 0x7ffb58195ee0, > d1 = 0x0, read_ahead = 0, msg_callback = 0x0, msg_callback_arg = > 0x0, hit = 0, param = 0x7ffb581933f0, > cipher_list = 0x0, cipher_list_by_id = 0x0, mac_flags = 0, > enc_read_ctx = 0x0, read_hash = 0x0, expand = 0x0, enc_write_ctx = > 0x0, write_hash = 0x0, compress = 0x0, cert = 0x7ffb58195ba0, > sid_ctx_length = 0, sid_ctx = '\000' <repeats 31 times>, session = > 0x7ffb58198100, > generate_session_id = 0x0,...... > > > > -- > *Gimhani Uthpala Kankanamge* > > > > -- > *Gimhani Uthpala Kankanamge* |
«
Return to OpenSSL - User
|
1 view|%1 views
| Free forum by Nabble | Edit this page |